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Abstract 



We introduce a generalization of the Fourier transform, denoted by S'c, on 
the isotropic cone C associated to an indefinite quadratic form of signa- 
ture (ni,ri2) on W 1 (n = n\ + ni: even). This transform is in some sense 
the unique and natural unitary operator on L 2 (C), as is the case with the 
Euclidean Fourier transform Jrh on L 2 (M n ). Inspired by recent develop- 
ments of algebraic representation theory of reductive groups, we shed new 
light on classical analysis on the one hand, and give the global formulas 
for the L 2 -model of the minimal representation of the simple Lie group 
G = 0{n\ + 1, W2 + 1) on the other hand. 

The transform S'c expands functions on C into joint eigenfunctions 
of fundamental differential operators which are mutually commuting, self- 
adjoint, and of second order. We decompose 7c into the singular Radon 
transform and the Mellin-Barnes integral, find its distribution kernel, and 
establish the inversion and the Plancherel formula. The transform "Sc re- 
duces to the Hankel transform if G is 0(n, 2) or 0(3, 3) ~ SL(4, K). 

The unitary operator $c together with multiplications and translations 
coming from the conformal transformation group C0{n\,n2) K ]R ni + n2 gen- 
erates the minimal representation of the indefinite orthogonal group G. Var- 
ious different models of the same representation have been constructed by 
Kazhdan, Kostant, Binegar-Zierau, Gross-Wallach, Zhu-Huang, Torasso, 
Brylinski, and Kobayashi-0rsted, and others. Among them, our model 
gives the global formula of the whole group action on the simple Hilbert 
space L 2 (C), and generalizes the classic Schrodinger model L 2 (R n ) of the 
Weil representation. Here, "Sc plays a similar role to . 

Yet another motif is special functions. Large group symmetries in the 
minimal representation yield functional equations of various special func- 
tions. We find explicit iT-finite vectors on L 2 (C), and give a new proof of 
the Plancherel formula for Meijer's G-transforms. 
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Chapter 1 

Introduction 



This book is a continuation of a series of our research projects [13 1 1561 Wf \ 
UHJ 09]. Our motif is to open up and develop geometric analysis of a single 
infinite dimensional representation, namely, the minimal representation tt of 
the indefinite (even) orthogonal group. 

This representation is surprisingly rich in its different models, through 
which we have cross- fertilization and interactions with various areas of math- 
ematics such as conformal geometry and the Yamabe operator, Fourier anal- 
ysis, ultra-hyperbolic equations and their conserved quantities, the Kepler 
problem, holomorphic semigroups, and analysis on isotropic cones. Among 
them, this book is devoted to the L 2 -model (Schrddinger model), for which 
the local formula was established in a previous paper [59] with B. 0rsted. 
The global formula of the whole group action is the subject of this book. 

We have limited ourselves to the very representation tt, although some of 
our results could be generalized to other settings by the ideas developed here. 
This is primarily because we believe that geometric analysis of this specific 
minimal representation is of interest in its own right, and might open up an 
unexpected direction of research bridging different fields of mathematics, as 
in the case of the Weil representation (e.g. [181 ESI Ell ESI EQ)- 

Bearing this in mind, we will not only 

• formalize our main results by means of representation theory, 
but also 

• formalize our main results without group theory. 

We have made effort to expound the theory in a self-contained fashion 
as much as possible. 
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For n = n\ + n<i, we denote by M ni ' n2 the Euclidean space M n endowed 
with the flat pseudo-Riemannian structure 

ds = dxi ~\~ ' ' ' ~\~ dx n _^ dx n _^_^_i • • • dx n , 

and define the isotropic cone C by 

C := {x E R n \ {0} : x\ + • • • + x 2 ni - x 2 ni+1 x\ = 0}. 

In this book, we will introduce the 'Fourier transform' Jc on the isotropic 
cone C for n even. This transform S'q is in some sense the unique and 
natural unitary operator on L 2 (C), as it is the case with the Euclidean 
Fourier transform 9^ on L 2 (JR. n ). 

Here is a brief guide to the three motivations of this book, with emphasis 
on the role of the unitary operator "Sq. 

The first motivation comes from analysis on the isotropic cone C itself. 
Different from non-isotropic hypersurfaces (e.g. hyperboloids) in IR ni,Tl2 , the 
restriction of ds 2 to C is degenerate, and we do not have a natural pseudo- 
Riemannian structure on C. Consequently, there is no natural single op- 
erator on C such as the Laplace-Beltrami operator. However, it turns out 
that there are commuting, self-adjoint, second order differential operators 
Pi, . . . ,P n that we call fundamental differential operators on C satisfying 

the algebraic relation P 2 -\ + P 2 X - P% 1+1 - ■ ■ ■ - P 2 = 0. Then, what 

we want is to understand how an arbitrary function on C (of appropriate 
class) is expanded into joint eigenfunctions of P±, . . . ,P n . 

We will find explicit joint eigendistributions for Pi, . . . , P n , and con- 
struct a (well-defined) transform, to be denoted by $c, by means of these 
eigenf unction. The transform S'q intertwines the multiplication by coordi- 
nate functions with the differential operators Pj. Moreover, we prove that 
we can normalize 3~c such that it is involutive, i.e. 3q = id and unitary. 
Thus, we establish its inversion formula and the Plancherel type theorem. 
It is noteworthy that the kernel function K(x,x') of S'c involves singular 
distributions (e.g. normal derivatives of Dirac's delta function with respect 
to a hypersurface) but yet that the operator 3~c is unitary in the general 
case where ni,ni > 1 and ni + 112 > 4. In the case ni = 1, 77,2 = 1 or 
(77-1,712) = (2,2), "Jc reduces to the Hankel transform composed by a (sin- 
gular) Radon transform. 

The second motivation comes from representation theory of real reduc- 
tive groups, in particular, from minimal representations. 

Minimal representations are infinite dimensional unitary representations 
that are the 'closest' to the trivial one-dimensional representation. The Weil 
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representation of the metaplectic group Mp(n,M), which plays a prominent 
role in the construction of theta series, is a classic example. Most minimal 
representations are isolated among the set of irreducible unitary represen- 
tations, and cannot be built up from the exsisting induction techniques of 
representation theory. 

A multitude of different models of minimal representations have been 
investigated recently by many people (see Sections 1.4 and 1.5). Each model 
known so far has its own advantages indeed but also has some disadvantages. 
For instance, the inner product of the Hilbert space is not explicit in some 
models, whereas the whole group action is not clear in some other models. 

A challenge to surmount that 'disadvantage' may turn up as a natural 
problem in other areas of mathematics. In order to give its flavor, let us 
consider two geometric models of minimal representations of the indefinite 
orthogonal group G = 0{n\ + 1, n,2 + 1): One is in the solution space to the 
Yamabe equation (conformal model), and the other is in L 2 (C) (Schrodinger 
model). 

In the conformal model, the whole group action is very clear, whereas 
the inner product is not. The problem of finding the explicit inner product 
was solved in the previous paper [17] as the theory of conserved quantities 
for ultra-hyperbolic equations, such as the energy for the wave equation. 

In the Schrodinger model L 2 (C), the unitary structure is clear, whereas 
the whole group action is not. The understanding of the whole group action 
was a missing piece of [47]. This problem is reduced to finding the gener- 
alization of the Fourier-Hankel transform on the isotropic cone C, namely, 
the above mentioned operator $c- By finding an explicit formula of S'c, 
we shall settle this problem. The role of "Jq in our minimal representation 
is in parallel to that of the Euclidean Fourier transform in the Weil 
representation, summarized as below: 



simple group 


Mp(n, R) 


0(m + l,n 2 + 1) 




(type C) 


(type D) 


Minimal representation 


Weil representation 


TT 


L 2 -model 


L 2 (W n ) 


L\C) 


(Schrodinger model) 






unitary inversion 


yj — 1 11 TV _ 

e 4 J Rn 





The third motivation comes from special functions. We note that the 
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isotropic cone C is so small that the group G = 0{n\ + l,rt2 + 1) cannot 
act on C continuously and non-trivially. This feature is reflected by the fact 
that the Gelfand-Kirillov dimension of the representation of G on L 2 (C) 
attains its minimum amongst all infinite dimensional representations of G. 
Thus, the representation space L 2 (C) is extremely 'small' with respect to the 
group G. In turn, we could expect a very concrete theory of global analysis 
on C by using abundant symmetries of the group G or its Lie algebra. 

It turns out that special functions in the Schrodinger model L 2 (C) arise 
in a somewhat different way from the well-known cases such as analysis 
on symmetric spaces (e.g. [31]) or its variants. For instance, the Casimir 
operator of K acts on L 2 (C) as a fourth differential operator. 

In this book, we encounter many classically known special functions (e.g. 
Bessel functions, Appell's hypergeometric functions, Meijer's G-functions, 
etc.). Special functions are a part of our method for the analysis of the 
minimal representation, and conversely, by decomposing the operator S'c 
we provide a representation theoretic proof of [inversion, Plancherel, . . . ] 
formulas of special functions including Meijer's G-functions. 

Encouraged by a suggestion of R. Stanton, we have decided to write a 
considerably long introduction. What follows is divided into three parts ac- 
cording to the aforementioned three motivations and new perspectives. In 
Sections I1.1H1.3| we state key properties of the involutive unitary operator 
Jc ° n L 2 (C) from analytic perspectives, in comparison with the well-known 
case of the Euclidean Fourier transform IFr™ on L 2 (W n ). Sections ll.4Hl.8l 
give representation theoretic perspectives, and we explain the role of "Sc in 
the Schrodinger model of the minimal representation of the indefinite orthog- 
onal group in comparison with the role of 3~iRn for the Weil representation. 
Thus, we compare Jc again with I^Rn, and correspondingly, the simple Lie 
algebra o(m + 1, 77-2 + 1) with sp(n,M). In Section [L9l we give a flavor of 
the interactions of the analysis on the minimal representations with special 
functions. 



1.1 Differential operators on the isotropic cone 

Consider an indefinite quadratic form on M. n = R ni+n2 : 

Q(x) := x\ + • • • + x 2 nx — a^ 1+1 — • • • — x^ 1+n2 . (1.1.1) 

Throughout the Introduction, we assume 77,1,77,2 > 1 and n = n\ + 77,2 is an 
even integer greater than two. (From Chapter 2, we will use the following 
notation: p = n\ — 1, q = ri2 — 1.) 
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Associated to the quadratic form Q, we define the isotropic cone 



C := {x G R n \ {0} : Q(x) = 0}, 
and endow C with the volume form d\i characterized by 

dQ A dfi = dx\ ■ ■ ■ dx n . 

Our object of study is the Hilbert space L 2 {C) = L 2 (C,d/j,) consisting 
of square integrable functions on C. 

A differential operator P on W 1 is said to be tangential to the submani- 
fold C if P satisfies 

*h\c = *h\c (P^i)\c = (Pih)\c (1.1-2) 

for any smooth functions ipi,tp2 defined in neighborhoods of C in W 1 . Then, 
we can 'restrict' P to C, and get a differential operator P\c on C. 
For instance, the following vector fields are tangential to C: 

n d 

E := > X{— — (the Euler operator), 

OXi 

i=i 

d d 

Xij := eiEjXi— - Xj-Q^- (1 < i < j < n), 

where we set ej = 1 or — 1 according as 1 < j < n\ or n\ + 1 < j < n. This 
is because the vector fields E and Xjj (1 < i < j ■ < n) are obtained as the 
differential of the conformal linear transformation group 

CO{Q) := {g G GL(n,R) : Q(ffa;) = cQ(x) ( v x G M n ) for some c > 0}, 

which preserves the isotropic cone C. 

Let M x, be the M-algebra of differential operators with polynomial 
coefficients (the W^eyZ algebra), namely, the non-commutative ring generated 
by the multiplication by x±, . . . , x n and the vector fields jj^, • • • , gf-- 

We denote by ~R[x, the subalgebra consisting of operators that are 
tangential to C. The multiplication by coordinate functions x^ clearly sat- 
isfies the condition (I1.1.2I) . Thus, we have seen 



x k, E, Xij G 



d_ 

' dx 



c 



[l<k<n, l<i<j< n). 



However, there exist yet other operators which are tangential to C, but are 
not generated by Xk,E,Xij in the Weyl algebra (see Remark l2.4.9p . 
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Among them are the fundamental differential operators of second order, 
to be denoted by Pi, . . . ,P n , which are defined by 

P j :=e jXj n-(2E + n-2)-^-. (1.1.3) 

Here, □ is the Laplace-Beltrami operator associated to Q, namely, 

-V d 2 d 2 d 2 d 2 d 2 

j^ 3 dx 2 ~ dx\ dx 2 nx dx 2 ni+1 dx 2 ' 

In the degenerate case rii = 712 = 1, our operators Pi and P2 take the 
following form: we set yi := x\ + X2, V2 '■= ^i — X2, 

d 2 d 2 

dyf dy$ 

see Remark l2.4.1Ul In general, these operators Pi, . . . ,P n satisfy the follow- 
ing properties (see Theorem 12.4. 1|) : 

PI PiPj = PjPi for any 1 < i, j < n. 

P2 Pj G R[x, J^f for any 1 < j < n. 

P3 The induced differential operators Pj\c ° n Co°(C) extend to self-adjoint 
operators on the Hilbert space L 2 (C). 

P4 (P 2 + ■ ■ ■ + Pi - P 2 i+1 P 2 )\ c = 0. 

P5 The Lie algebra generated by xi, Pi (1 < i < n) contains the vector 
fields E, Xij (1 < i < j < n). 

From now on, we simply write Pj for Pj\c- Thus, we have commuting 
self-adjoint, second-order differential operators Pi, ...,P n on L 2 (C). 
We are brought naturally to the following: 

Problem 1.1.1. 1) Find joint eigenf unctions of the differential operators 
Pi,...,P n on the isotropic cone C . 

2) Given a function f on C, find an explicit expansion formula of f into 
joint eigenf unctions of Pi , . . . , P n . 
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1.2 'Fourier transform' 3^c on the isotropic cone C 

In this book, we shall give a solution to Problem ll.l.ll by introducing a 
unitary operator 3~c on L 2 (C). 

To elucidate the operator $Ci let us consider first much simpler operators 



in place of Pj. Then, pi,...,p n form a commuting family of differential 
operators which extend to self-adjoint operators on L 2 (M n ). Analogously 
to Problem consider the question of finding the explicit eigenfunction 
expansion for the operators pi, . . . ,p n . Then, as is well-known, this is done 
by using the (Euclidean) Fourier transform £F = ST^n on R n . In what follows, 
we normalize 9^ as 



where (x, £) = Y17=i an< ^ ^ x = ' ' ' ^ x « - We n °te that the signature 
of the power here is opposite from the usual convention. Obviously, the 
kernel 



of the Fourier transform S^n is real analytic on the direct product space 



We recall the following key properties of the Euclidean Fourier transform: 





(1.2.1) 




(2vr)2 



Fl p j k(x,€)=t j k(x,£). 



F2 k(x,0=H£,x). 



F3 J K n(qf (R n )) C C°°(R n ) nL 2 (i n ). 

F4 GFrti extends to a unitary operator on L 2 (IR n ). 

F5 J K n O Xj = Pj O Jjgn, 



3~r™ ° Pj = ° Stj™ ■ 
F6 (3^ n u)(x) = u(-x), = id - 
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F8 3~Rn gives an automorphism of each of the following topological vector 
spaces: 

S(M n ) C L 2 (R n ) C S'(K n ). 

Here, F7 gives the plane wave expansion of the Fourier transform by means 
of the Radon transform R defined by 

Ru(£,t) := / u(x)5((x,g) -t)dx. 

In Ij8l we denote by S(M n ) the space of rapidly decreasing C°°-functions on 
W 1 (the Schwartz space endowed with the Frechet topology), and by §'(R n ) 
the dual space consisting of tempered distributions. 

F|4] is the Plancherel theorem, and F8 gives the Paley-Winer theorem 
for the Schwartz space S(M n ) (and its dual S'(]R n )). By Fl and F2J the 
inversion formula 

f(x) = J R n o 3^;f(x) 

= [ (3$f)(0Hx,0dt 

gives an expansion of a function / into joint eigenfunctions k(x,£) of the 
commuting self-adjoint operators pj (1 < j < n). 

Moreover, the property F5 characterizes the operator S'^n up to scalar. 
We pin down this algebraic statement in two ways as follows: 

Proposition 1.2.1. Let A be a continuous operator on L 2 (M. n ) satisfying 
the following identities: 

A o Xj = pj o A, Aopj = -Xj o A (1 < j < n). (1-2.2) 

Then, A is a scalar multiple of S'^n . 

Proposition 1.2.2. Let A be a continuous operator on L 2 (W n ) satisfying 
A o XjXj = piPj 0^4,^4 ptfj = XiXj o A (1 < i,j < ri). (1.2.3) 
Then, A is of the form A = a3~Rn + 63^n for some a,b G C. 

Here, (1.2.3) is obviously a weaker condition than (1.2.2). 
We did not go into details about the domain of definition for (|1.2.2p 
and (jl.2.3p in the above propositions. The domain could be D := {/ £ 
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L 2 (R n ) : Xjf,pjf G L 2 (R n ) (1 < v j < n)}, on which we regard the identities 
Axjf = PjAf those of distributions in the case of Proposition ll.2.11 Likewise 
for Proposition 11.2.21 

Intertwining characterization of $c 

Back to the setting in Section [TTT| we consider the differential operator Pj 
(of second order!). Then, it turns out that the intertwining relation between 
Pj and the multiplication by the coordinate function Xj again characterizes 
our operator 9~<7 up to scalar: 

Theorem 1.2.3 (see Theorem [2~5.4p . 1) There exists a unitary operator S'c 
on L 2 (C) satisfying the following relation: 

A o Pj = Axj o A, Aoxj = APj oA (1 < j < n). (1.2.4) 

2) Continuous operators A on L 2 (C) satisfying (|1.2.4p are unique up to 
scalar. In particular, any such operator A is a scalar multiple of the 
unitary operator $c, and A 2 is a scalar multiple of the identity operator. 

Joint eigendistributions on C 

Next, we consider the system of differential equations on C: 

Pj^ = 4(jip (1 < j < n). (1.2.5) 

The coefficient 4 in the right-hand side is just for simplifying later notation. 

We shall deal with solutions in an appropriate class of distributions on C 
(the dual space L 2 (C)~°° of smooth vectors L 2 (C)°°, see (|2.5.9p ) and prove 
the following: 

Theorem 1.2.4 (see Theorem EjT5| • Fix £ = • • • , £n) 6l"\ {0}. 

1) If Q{C) 7^ 0, then any distribution ip on C satisfying (1.2.5) is zero. 

2) //£ S C, then the solution space of (1.2.5) in L 2 {C)~°° is one- dimensional. 

The first statement is an immediate consequence of P4. By the explicit 
formula given in Theorem 11.3.1} we shall see that the unique solution in 
Theorem 11.2.41 (2) is not real analytic if n\^ni > 1. 

Abstract properties of $c 

We will prove in this book that the distribution solution ip(x) in Theorem 
11.2.41 (2) can be normalized depending on £ G C, which we denote by K(x, £) 
for now, in such a way that the following key properties are fulfilled: 
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Kl For each fixed £ £ C, -f£T(-,£) is a distribution solution on C to (|1.2.5p . 
K2 A'(x, f ) = K{£ distribution on C x C. 

K3 We define 

(3c/)(0:= / K(x,Of(x)dfi(x) for / e Cg°(C). (1.2.6) 
Then, (1.2.6) is well-defined, and we have a linear map "Sq : Cq°(C) — > 

c°°(C)r\L 2 {C). 

K4 9"c extends to a unitary operator on L (C). 

K5 o 4xj = P, o y c , 
J c o /> = 4£j o J c . 

K6 gf, = id. 

K7 ? c u(0 = J R f(tW(Z,t)dt 

K8 $c gives the automorphism of each of the following topological vector 
spaces: 

L 2 (C)°° C L 2 (C) C L 2 (C)-°°. 

These properties K1-K8 are stated in parallel to the Euclidean case FI- 
FO In K0 31 is the (singular) Radon transform on the isotropic cone C 
which will be defined in (|1.2.7|) . and ^(t) is a distribution on R which will 
be defined in Theorem 11.3.11 We note that the transform in K7 by ^(t) 
collapses to the Hankel if ni = 1. 

In K[8j we have the following inclusive relation 

C °°(C) C L 2 (C)°° C L 2 (C) C L 2 (C)-°° C D'(C) 

as in the Euclidean case (see F8): 

C^°(M n ) C S(M n ) C L 2 (M n ) C S'(R n ) C D'(M n ). 

In summary, K4 is a Plancherel type theorem of $c on L 2 {C), K6 gives 
its inversion formula, K7 expresses $c by 'plane wave' decomposition, and 
K|8] gives a Paley-Winer type theorem for the 'Schwartz space' L 2 (C)°°. 

The above formulation brings us naturally to the following program: 
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Program 1.2.5. Develop a theory of 'Fourier analysis' on the isotropic 
cone C by means of $c ■ 



We expect that this program could be enhanced by a solid foundation 
and concrete formulas of the transform "Sq. 

For this, the first step is to find explicit formulas for the (normalized) 
joint eigenfunctions K(x,£). We prove that they are given by means of 
Bessel distributions (see Theorem ll.3.ip . In particular, K2 follows readily 
from the formulas. The properties Kl, K4l K[5| and K6 will be proved in 
Theorem 12.5.21 based on a representation theoretic interpretation that 
is the 'unitary inversion operator' on L 2 (C) for the minimal representation 
of the indefinite orthogonal group 0{n\ +1,712 + !)• By K6; we get the 
inversion formula just as S^ 1 = $c, which gives an explicit solution to the 
problem of joint eigenfunction expansions (see Problem ll.l.l|) : 



The kernel K(x,£) is not locally integrable but is a distribution in gen- 
eral. To see the convergence of the right-hand side (|1.2.6|) . we note that 
K(x,£) depends only on (x,£) = Y^i=i x i£i ( see Theorem 11.3.1ft . This fact 
leads us to the factorization K7 through the (singular) Radon transform 
3£ on the isotropic cone C, which is defined by the integration over the 
intersection of C with the hyperplane 



For a quick summary of the transform 31 (see Chapter 5jfor details), we 
identify a compactly supported smooth function / on C with a measure fdfi. 
It is a tempered distribution on W l (n > 2). Then, the Radon transform Ji 
of fdfj, is defined by 



for <E (K n \ {0}) x (R\ {0}). The point here is that the integration is 
taken over the isotropic cone. In other words, t) is obtained by the in- 

tegration over submanifolds which are generically of codimension two in K n . 
Consequently, Jlf(£,t) satisfies the ultra-hyperbolic differential equation of 
the ^-variable: 



f(x)= [ (? c f)(0K(x,0dK0 



Jc 



{x G R n : (x,C) =t}. 




(1.2.7) 




(1.2.8) 
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Next, in order to see the regularity of at t = 0, we fix £. Then, 

the intersection of the isotropic cone C with the hyperplane {x G M. n : 
(x,£) = t} forms a one parameter family of submanifolds of codimension 
two for f / 0, which have singularities at t = 0. Accordingly, the Radon 
transform is not of C°° class at t = even for / G Cg°(C). The 

regularity of t) at i = is the principal object of the paper [53] , 

where it is proved that 0lf(^,t) is times continuously differentiable at 

t = 0. Here, [x] denotes the greatest integer that does not exceed x. This 
regularity is exactly sufficient for what we need to prove that the singular 
integral (|1.2.6p makes sense for / G Cq°(C). See Section HT21 for details. 

The reverse direction, namely, the application of our results on Jq to 
the results on the singular Radon transform includes: 

Corollary 1.2.6. Any compactly supported smooth function f G Cq°(C) 
can be recovered only from the restriction of the Radon transform 0?/(£,i) 
toC xl. 

Underlying algebraic structures 

The underlying algebraic structure of Propositions 11.2.11 and 11.2.21 and 
Theorem 11.2.31 will be revealed by the Lie algebras generated by the differ- 
ential operators in each setting as follows: 

In Proposition 11.2.1] the Lie algebra generated by pi,Xi (1 < i < n) is 
the Heisenberg Lie algebra. 

In Proposition I1.2.2] the Lie algebra generated by piPj,XiXj (1 < i,j < 
n) is the symplectic Lie algebra sp(n,M). 

In Theorem 11.2.31 the Lie algebra generated by Pi,Xi (1 < i < n\ + 722) 
is the indefinite orthogonal Lie algebra o{n\ + l,ri2 + 1). 

These actions of the Lie algebras lift to unitary representations of the 
corresponding Lie groups: On L 2 (M n ), the Schrodinger representation of 
the Heisenberg group, and the Weil representation of the metaplectic group 
Mp(n, K), namely, a double cover of the symplectic group Sp(n,l&) (see Sec- 
tion [L3J; on L 2 (C), the minimal representation of the indefinite orthogonal 
group 0(ni + l, n2 + l) (see Section IT75|) . In Sections ll.4Hl.8t we shall discuss 
some perspectives from representation theory. Before entering representa- 
tion theory, we continue an account from the viewpoints of analysis in the 
next section. 
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1.3 Kernel of $c and Bessel distributions 



In this section, we give an explicit formula of the kernel distribution K(x,£) 
of the transform 3^c on L 2 (C), and observe the similarities to and differ- 
ences from the kernel k(x,£) = (2ir)~2 eV^-TfoS) Q £ ^ e Euclidean Fourier 
transform S^n on L 2 (IR n ). 

Here is the highlight of this book. 

Theorem 1.3.1 (see Theorem 15.1.11) . Suppose n = m + n<i is even, > 2. 
The unique unitary operator 3~c on L 2 (C) in Theorem \1. 2. 3\ is given by the 
distribution kernel K \x , £) := ^((x, £)), where 

n^ni-l) n -2 

¥(t) = 2(-l) a vr" — 



x < 



$t=±( t ) */ min(ni,n 2 ) = 1, 

2 

^ n-4 (i) */ n i> n 2 > 1 a^e 6o/j/i odd, 
~ 2~ 

(t) if n\,n2 > 1 are both even. 



2 



As for the normalization of a scalar constant in the above theorem, we note 
that the intertwining property (1.2.4) determines K{x,£) up to scalar, and 
moreover K6 determines K(x,£) up to signature. The signature is taken 
to be compatible with the action on the Schrodinger model of the minimal 
representation of 0{n\ + I,n2 + 1) which will be discussed in Section [L8l 

In Theorem 1 1 . 3. 1|. ( , ) denotes the standard (positive definite) inner 
product on IR n . ^^(i) and *S? m (t) are tempered distributions on M, 

defined below in (fTBTT]) . ([LIT!?]) , and ([TIP]) , respectively. 

Bessel distributions 

Let J v (x), Y u (x) and ^(z) be the (modified) Bessel functions (see Ap- 
pendix E2]). We use the following notational convention: 

fit \-=l m {t>0) fit )-=l° ( '" 0) 
n+J ' \o (t<0), \/(|t|) (t<0), 

for a function (or a 'generalized function') f(t) on M. Then, and 
\I/ m in Theorem 11.3.11 are the distributions on M. given by 



$+(t) :=(2t), 2 J m (2y^:), (1.3.1) 




«+(t) := (»)+ 2 J m (2v^) - ^ ^_ fc) , ^(t), (1.3.2) 

_m 2f — l) m+1 -S 

* m (t) := (2t) + 2 y m (2 v / 2t7) + -i ^ (2t)_ 2 K m (2y/2tl). (1.3.3) 

7T 
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Here, (2t+) 2 J m (2- v /2i + ) makes sense as a locally integrable function on M. 

_ m m 

On the other hand, (2t) + 2 F m (2^/2t7) and (2i)_ 2 if m (2 v / 2tl) are defined 
as regularized distributions, and ^ m (t) have the following singularity: 

_Y m t~ k (k — 1)\ 

f m (t) = (locally integrable function) H N — r- — — . 

7T ' 2 k (m — k)\ 

We shall say that , and ^ m are Bessel distributions. 

Integral expressions and differential equations 

The Bessel distributions <£+(i), \l/+(£), and ^^(^) are rea, l analytic on 
E \ {0}, and satisfy the following differential equation: 

or equivalently, 

(9 2 + m# + 2t)V = 0, 

where := t4r. Furthermore, all of the three solutions satisfy the following 
asymptotic behavior 

„ , 2m+l , 

ty(t)=0(t 3—) ast^+oo. 

In this book, we adopt an alternative definition of and ^ m in 

Section 16.21 by means of the Mellin-Barnes type integral for distributions. 
The expressions ([1.3.1)1 — ( [1.3.3P will be explained there. Another (slightly 
different) expression of 3>+, Vt^, and \P m by means of 'normalized' Bessel 
functions J m , K m , and Y m is given in Remark 16.2.31 

Chapter [6| is devoted to these Bessel distributions. We shall discuss their 
integral formulas and differential equations. 

Support of the kernel K(x,£) 

Unlike the kernel k(x, £) = (2ir)~2 ev~*( x & for the Euclidean Fourier 
transform 3~iRn , our formula in Theorem 11.3.11 shows that the support of the 
kernel K(x,£) for "Jq differs according to the signature (n\,ri2). 

To see this, we set the 'half space of the direct product manifold C x C 

by 

(C x C)+ := {(x,0 € C x C : (x,£) > 0}. 
Then, we have the following mysterious phenomenon: 
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supp-fC(x, £) 



Corollary 1.3.2. The kernel K(x,£) of the unitary operator $c on L 2 (C) 
satisfies 

(C x C)+ if n\,U2 both odd, 
C x C if n± , nz both even. 
In particular, 

suppiT(x,£) ^CxC if ni,rt2 both odd. 

If 77,1 = 1 or ri2 = 1, the isotropic cone C is the light cone in the rela- 
tivistic cosmology, which splits into two connected components, namely, the 
forward light cone C+ and backward light cone C_ , and correspondingly, we 
have a direct sum decomposition as Hilbert spaces: 

L 2 (C) = L 2 (C+)©L 2 (C_). (1.3.4) 

Then, $c leaves L 2 (C + ) and L 2 (C_) invariant, respectively (see 



This gives a geometric explanation of Corollary 11.3.21 because 

(CxC)+ = (C+xC + )U(C_xC_) (1.3.5) 

in this case. On the other hand, if n\,ri2 > 1, then C is connected and we 
do not have a natural decomposition of (C x C)+ like (|1.3.5j) . Moreover, 
the representation ir of the indefinite orthogonal group 0(n\ + 1, n,2 + 1) 
on L 2 (C) (discussed later) stays irreducible when restricted to the identity 
component SOo(ni + 1, n2 + 1). Nevertheless, the support of K{x, £) is half 
the space of C x C when both ni and ri2 are odd integers greater than one. 
We do not see its intrinsic reason. 

Singularities of the kernel K(x, £) 

Another distinguishing feature of our kernel K(x,^) for S~c is that it is 
not real analytic on C x C, whereas the kernel k(x,£) = (27r)~ ie^~^( x, & for 
the Euclidean Fourier transform GFr™ is obviously real analytic on M n x M n . 

Among the three Bessel distributions introduced in (|1.3.1|) - (|1.3.3|) . $^(t) 
is a locally integrable function on M, whereas ^^(i) and *$> m (t) (m > 1) 
are not. The singular part of the distribution ^P^j(t) is given as a linear 
combination of the Dirac delta function S(t) and its Ith derivative 5^ l \t) 
(I = 1, 2, . . . ,m — 1). The singular part of the distribution Vl/ m (t) is given by 
a linear combination of the distribution t~ k (k = 1,2, ... , m) (see Theorem 

Some readers might wonder why the kernel function of a unitary operator 
involves such singularities. So, let us examine to which extent the regular- 
ity of the kernel K(x, £) is required from the general theory of functional 
analysis. 
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By the Schwartz kernel theorem, any continuous operator T : L 2 (C) — > 
L 2 (C) is expressed as 

(Tf)(0= I K(x,Of(x)d t x(x) 
Jc 

by some distribution kernel K(x,^) E D'(C x C). Here, we regard K(x,£) 
as a generalized function by using the measure dfi on C (sec [25J). 

If T is a Hilbert-Schmidt operator, then K £ L 2 (C x C). If T is the 
identity operator, then i*T is Dirac's delta function 5(x — £). In general, 
the continuity of T forces any such K to be at most (dimC + 2) times 
derivatives of a locally integrable function on C x C (see [70 \ pp. 296-299] 
for the argument using the Sobolev space theory in the compact torus case). 

'Laurent series expansions' of Bessel distributions 

We end this section with an interesting observation on 'Laurent series 
expansions' of Bessel distributions and 

= £ w i-ii^ + iv (L3 - 6) 

^ r(m + j + i)r(j + 1) 

J= j=-oo v w ^ 

We note that $^ arises as the kernel for S'c when (711,712) = (2m+3, 1), and 
that Vl/+ arises when m, ri2, > 3 are both odd and m + n2 = 2m + 4. The 
first formula (|1.3.6p is a usual Taylor expansion. But the second formula 
(1.3.7) involves negative terms, for which we need a justification. For this, 
we think of th_ as a distribution meromorphically dependent on A, and then 
we get 



r(i + A) 

by (|7.1.ip . Therefore, we have 
-l) A 2 A ti 



r(m + A + l)r(l + A) 



(m + l<k). 



In this sense, the series (|1.3.7p contains only finitely many negative terms, 
and is equal to (|1.3.2p . 
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1.4 Perspectives from representation theory — find- 
ing smallest objects 



The philosophy of analysis and synthesis asks for the understanding of the 
smallest objects and of how things are built from these objects. 

For a Hilbert space "K over C, we denote by U(!K) the group consisting 
of unitary operators on "K. By a unitary representation on a Hilbert space 
"K of a topological group G, we mean a group homomorphism 

7T : G -» U{%) 

such that G x % — ► !K , (p, u) i— > 7r(g)v is continuous. 

The 'smallest objects' of unitary representations are irreducible unitary 
representations. By a theorem of Mautner and Teleman, any unitary repre- 
sentation of a locally compact group G (e.g. a Lie group) can be decomposed 
into the direct integral of irreducible unitary representations of G, see |78] , 
The classification of irreducible unitary representations of Lie groups has 
been a long standing unsolved problem since 1940s, originally arising from 
quantum mechanics. 

The 'smallest objects' of Lie groups consist of simple Lie groups such 
as SL(n,M), 0(p,q), and Sp(n,M), and one-dimensional abelian Lie groups 
such as M and S . Loosely speaking, a theorem of Duflo [11] asserts that all 
irreducible unitary representations of general (real algebraic) Lie groups are 
built up from those of simple Lie groups. 

However, irreducible unitary representations of simple Lie groups are 
not fully understood despite huge efforts for many decades and also sig- 
nificant results. Among them, powerful algebraic machinery including the 
theory of cohomological induction has been largely developed in 1980s by 
Zuckerman, Vogan, Wallach and others \77\ I78|. As a result, the problem 
of classifying irreducible unitary representations has been focused on those 
representations of simple Lie groups that cannot be 'induced up' from other 
representations. Such representations may be regarded as 'atoms' of unitary 
representations of Lie groups, and they are still mysterious creatures. See 
Vogan [77], for example, for a discussion on how to understand them as the 
theory of 'unipotent representations' of reductive Lie groups. 

'Minimal representations' are the simplest, infinite dimensional 'unipo- 
tent representations'. There has been active study on minimal represen- 
tations, mostly by algebraic methods since 1990s (see Gan and Savin [21] 
and J.-S. Li [52] for surveys both in the real and in the p-adic fields). In 
contrast to these existing algebraic approaches, new geometric analysis of 
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the minimal representation of the simple Lie group 0(p, q) is a motif of this 
book. We will discuss minimal representations in the next section followed 
by Sections ll.6til.8l in more details. 

1.5 Minimal representations of simple Lie groups 

To formulate the 'smallness' of an irreducible representation ir of a simple Lie 
group G, algebraic representation theory usually appeals to the 'largeness' of 
the annihilator Ann(-7r) in U(qc) of the differential representation dir. Here, 
U(qc) is the universal enveloping algebra of the complexified Lie algebra 
0C = ®M C. We recall: 

Definition 1.5.1. An irreducible unitary representation n of a simple Lie 
group G is a minimal representation if the annihilator Ann(7r) is equal to 
the Joseph ideal JM\3^ ofU{Q C )- 

The Joseph ideal Ann(-7r) is a completely prime ideal whose associated 
variety 'V(Annvr) is the closure of 0^ in . Here, 0^ in is the complex mini- 
mal nilpotent orbit in g c for the coadjoint representation. Therefore, the 
Gelfand-Kirillov dimension of it, to be denoted by DIM7T, satisfies 

DIM tt = ^ dime 0£ in! 

and in particular, ir has the smallest possible Gelfand-Kirillov dimension. 

The Weil representation w of the metaplectic group Mp(n, E) is a classic 
example of minimal representations (to be precise, the Weil representation 
decomposes into a direct sum of two irreducible representations, both of 
which are minimal representations), and DIMzu = n. 

The study of minimal representations of other reductive groups is rel- 
atively new, and it is only in the last two decades that various models of 
minimal representations have been proposed and studied extensively (see 

^ [Zi [131 [13 1211 12ZI 133 1^ 1^ BZl BBl 1^ 1501 1521 1^1 ISD] for instance). 

We note that not every reductive Lie group admits minimal represen- 
tations. For instance, the minimal nilpotent orbit 0^ in of SO(m,C) has 
the dimension 2(m — 3), and it was found by Howe and Vogan (see [76] ) 
that there exists no representation of any covering group of SOq(p, q) (or its 
finite extension) whose Gelfand-Kirillov dimension equals p + q — 3 if p + q 
is odd and p, q > 3. This is the primary reason that we dealt with the case 
n = n\ + 712 is even in Sections ll.ltil.3 . Here, p = n\ + 1 and q = ri2 + 1- (It 
is another story that there exists an 'infinitesimally unitary' and 'minimal' 
representation of the Lie algebra so(p, q) for p + q odd, see [35].) 
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In accordance with the philosophy of the orbit method advocated by A. 
Kirillov, B. Kostant, and M. Dimo, irreducible unitary representations of 
Lie groups G are supposed to be attached to coadjoint orbits in g*. This 
works perfectly for simply-connected nilpotent Lie groups. However, for 
simple Lie groups, the orbit method does not work very well, though the 
orbit method still gives an approximation of the unitary dual. In particular, 
it is not known how to attach unitary representations to nilpotent orbits of 
simple Lie groups. 

The reverse direction is easier. As we have seen at the beginning of this 
section, we can attach nilpotent orbits to admissible representations tt (in 
particular, to irreducible unitary representations): 

vr ~* Ann(vr) ^ V(Ann(vr)). (1.5.1) 

There are also several ways to attach (a union of) real nilpotent orbits in g* 
to admissible representations tt such as 

g* D WFn(vt) : wave front set by Howe, [M] 

g* D WF(7r) : asymptotic support of the character Trace tt, [3] 

3 Ass(7r) : associated variety of the underlying (gc, if)-module of tt, [75] 

Here, g = 6 + p is a Cartan decomposition and gc = 6c + pc is its complex- 
ification. WFn(vt) and WF(7r) are a union of nilpotent orbits of G in g*, 
and Ass(-7r) is a union of nilpotent orbits of Kq on p£. It was conjectured 
by Barbash and Vogan [3j, and proved by Schmid and Vilonen [M] that 
WF(-7r) corresponds to Ass(-7r) via the Kostant-Sekiguchi correspondence. 
For a minimal representation tt, these invariants are contained in the clo- 
sure of the intersection g* n 0^ in , p c n 0^ in , respectively. Here, 0^ in is the 
complex minimal nilpotent orbit in g£», and we identify gc with gj. by the 
Killing form. 

Conversely, it is much more difficult to construct irreducible unitary 
representations from nilpotent orbits in general. If the complex minimal 
nilpotent orbit 0^ in has a non-empty intersection with g* , then 0^ in n g* 
consists of equi-dimensional Ad*(G)-orbits, namely, minimal nilpotent orbits 
in g* . An optimistic picture is that minimal representations are attached to 
such orbits, however, this is false in general (see [72] for the s[(3, R) case). In 
this direction Brylinski and Kostant [7] constructed minimal representations 
from Ass(-7r) on the (gc, -RT)-module level, but both the Hilbert structure and 
the whole group action are not given globally, depending on each .fT-type. 

In this book, we deal with G = 0{n\ + l,n2 + 1) (ni,ri2 > l,ni + n<i 
even > 4), where m i n := O^in H g* is a single G-orbit, namely, a mini- 
mal nilpotent orbit. Our minimal representation is realized on the concrete 
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Hilbert space L 2 (C). On the other hand, the isotropic cone C becomes a La- 
grangian submanifold of m j n which is endowed with the natural symplectic 
structure. In this sense, our Schrodinger model L 2 (C) may be viewed as a 
geometric quantization of the real minimal coadjoint orbit m i n . Our main 
theorem enables us to give the whole group G-action on L 2 {C) explicitly 
and independently of if-types. 

1.6 Schrodinger model for the Weil representation 

In Section \1.7\ we shall discuss the minimal representation of G = 0{n\ + 
1, ri2 + 1), which is a simple group of type D if n\ + rii is even, > 2. In this 
section, we recall the best understood minimal representation of a simple 
group of type G, that is, the Segal-Shale- Weil representation w, simply, the 
Weil representation, or sometimes referred to as the oscillator representa- 
tion, or harmonic representation, of the metaplectic group Mp(n,M). Here, 
Mp(n,M) is the twofold cover of the real symplectic group Sp(n,M). Let £o 
denote the (unique) non-trivial element in the kernel of the homomorphism 
Mp{n, K) — > Sp{n, K). That is, we have an exact sequence of Lie groups: 

1 -» {e,fo} -» Mp(n,R) -► Sp(n,R) -► 1. 

We will set G' := Mp(n,R). 

Among various realizations of the Weil representation (see [52\ §3] for 
a brief survey), the Schrodinger model gives a realization of the Weil rep- 
resentation w on the Hilbert space L 2 (M n ). Since our model (7r,L 2 (C)) of 
the minimal representation of G has a strong resemblance to (w, L (R n )) of 
G', we list some important aspects of the Schrodinger model of w (see e.g. 

[IE! [22 EH]): 

Cl The representation is realized on a very concrete Hilbert space, that is, 
L 2 (R n ). 

C2 The restriction of w to the Siegel parabolic subgroup Psicgci is still 
irreducible. The restriction tJ7p Sie ol has a relatively simple form (trans- 
lations and multiplications by unitary characters). 

C3 The infinitesimal action dm of the Lie algebra sp(n,M) is given by dif- 
ferential operators of at most second order. 

C4 There is a distinguished element w' of G' that sends -Psiegei to the op- 
posite parabolic subgroup. The corresponding unitary operator vj(w' q ) 
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on L (R n ) is equal to e * 9tr«- Correspondingly to the fact that 
(w' ) 4 = £o and ro(£o) = — id, the Fourier transform S^n is of order 
four. 

We write Sp(n, R) in the matrix form as 



Since G' is generated by -Psiegei and it;^, C2 and C4 determine the action 
of G' on L 2 (M n ) (see [59] for an explicit formula of the whole group G'-action 
on L 2 (R n )). As for 031 if X £ psiegei (the Lie algebra of Psiegei), then dw(X) 
contains a differential operator of second order, and is not given by a vector 
field. This reflects the fact that G' acts only on L 2 (R n ), and not on R n . 

In various places of this book (usually, as Remarks), we will compare 
our results with the corresponding results for the Weil representation. We 
will see that the case n\ = 1 or i%2 = 1 in our setting G = 0(n\ + 1, 712 + 1) 
has similar features to the case of the Weil representation in both analytic 
and representation theoretic aspects, and that the general case ni,ri2 > 1 
often provides new analytic features. 

1.7 Schrodinger model for the minimal represen- 
tation of 0(p, q) 

In this section, we consider the indefinite orthogonal group 



V u 1 <iJ 

four. Then, there exists a distinguished irreducible unitary representation 
7T of G with the following properties: 

Ml -/r is still irreducible when restricted to the identity component SOq(p, q) 
of G if and only if p, q > 2. 



Sp(n,R) = {g€ GL(2n,R) : * 5 J n5 = J n } 




Then, down to Sp(n,R) ~ Mp(n,R) mod {e,£o} 




where /, 
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M2 If p = 2 or q = 2, then tt is a direct sum of an irreducible unitary 
highest weight representation and a lowest weight representation. 

M3 tt is spherical (i.e. has a non-trivial vector fixed by K = 0(p) x O(q)) 
if and only if p = q. 

M4 7r is a minimal representation in the sense of Definition ll.5.1l if p + q > 6 
(see PEO]). 

M5 WF(vr) = min = (0^ n fl*), Ass(tt) = 0^ n p* c , and DIM(vr) = 
p + q -3, (see @7]). 

The above properties M2 and M3 show that our minimal representation tt 
is neither a spherical representation nor a highest weight representation in 
the generic case where p / q, p, q > 2, and p + q > 6. 

Various realizations of the minimal representation tt have been proposed 
so far by a number of people. For example, Kazhdan in [39J, and Kostant 
in [50] for p = q = 4, and Binegar and Zierau [6] for general p, q > 2, 
constructed tt as a subrepresentation of a maximally degenerate principal 
series representation (see also Howe and Tan |36] for a full discussion on 
its composition series); Zhu and Huang [80] constructed tt as the theta cor- 
respondence of the trivial one-dimensional representation of SL{2, M) for 
the reductive dual pair 0(p,q) • SL(2,M) C Sp{p + q, E), see also [51] ; and 
Kobayashi and 0rsted [U] constructed -tt as the solution to the Yamabe 
equation (conformal model); in [49] the Schrodinger model (an L 2 -model) 
of tt. Yet another construction has been proposed in Brylinski and Kostant 
0, Gross and Wallach [27], and Torasso [73]. 

Among various realizations of the minimal representations, our concern 
is with the Schrodinger model realized on L 2 (C). Here, C is the isotropic 
cone in R ni+n2 = W 1 discussed in Sections ll.lfll.3|, and the relation between 
the above parameters p and q is given by 

p = m + 1, q = n 2 + l, n= p + q-2. 

The conformal linear transformation group CO(Q) (c GL(ni +7i2,M)) 
acts on the isotropic cone C (c M ni+n2 ), and then induces a unitary rep- 
resentation on L 2 (C) (see f|2.3.10j) - (|2.3.12l) for a concrete formula). Much 
more than that, this action on L 2 (C) can be extended to a unitary repre- 
sentation tt of the indefinite orthogonal group G = 0{n\ + 1,^2 + 1). To 
explain its idea, we also recall another geometric model, namely, the confor- 
mal model. 
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The conformal model of the minimal representation tt is realized, for 
example, in the solution space to the ultra-hyperbolic equation 

j¥_ d 2 d 2 \ 

\dx 2 dx 2 ni dx 2 ni+1 dx 2 J U 

in S'(R n ) (tempered distributions), and also in the solution space 

A 5 » lx5 n 2 « = onS ni xS n2 

where A S "i x s n 2 = A s »i - A s « 2 - ( HJ ^-) 2 + ( na ^-) 2 is the Yamabe oper- 
ator on the direct product manifold S ni x S n2 equipped with the pseudo- 
Riemannian structure of signature (m,^)- These two models are isomor- 
phic to each other by the general theory of conformal geometry, and the 
intertwining operator is given by the 'twisted pull-back' ^* of the conformal 
map ^ : R ni + n2 — ► S ni x S n2 , the inverse of the stereographic projection 
(see |43| for an elementary account). Then, taking the Fourier transform 
3~Rn of the conformal model, we get the Schrodinger model L 2 (C). 

The intertwining operator T : L 2 (C) — > Sol(As n ixS n 2) is defined in 
Section [2.21 so that the following diagram commutes: 

Conformal model 
(if -picture) 

SoZ(Asn 1XiS <n 2 ) 




SoZ(D R ~i,n 2 ) ► L 2 (C) 

Conformal model Schrodinger model 

(iV-picture) 



Diagram 1.7.1 

We remark that the isotropic cone C is defined as a hypersurface in 
M n = R ni + n2 ; and the group G = 0{n\ + 1, n 2 + 1) cannot act (non-trivially) 
on C. (In fact, any (non-trivial) G-space is of dimension at least n\ + n 2 = 
dimC + 1.) 

In the L 2 -model L 2 (C) of the indefinite orthogonal group G, the action 
of a maximal parabolic subgroup 

~ Z 2 ■ CO(Q) x 
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on L 2 (C) is of a simple form (see f)2.3.10|) — f)2.3.13|) for definition). This 
parabolic subgroup P max plays a similar role of the Siegel parabolic subgroup 
-Psiegeh an d analogous results to the properties CjlJ C|2| and C[3| hold (see 
Section I2.3P . If we set 

wo := J ni+ i jna +i, 

then wo sends P max to the opposite parabolic subgroup _p max ) and G is 
generated by wq and P max . 

In light of the Bruhat decomposition G = P max II P max woP max , we can 
get directly the concrete form of the action of the whole group G once we 
know tt(wo) explicitly. 

In the degenerate case {n\, 712) = (2, 0), G = 0(3, 1) is locally isomorphic 
to 5*1/(2, C) acting on M 2+0 (~ C) as Mobius transforms (linear fractional 
transforms), and wq acts on C as the conformal inversion 

C — > C, z • — ► — . 

z 

Although (711,712) = (2,0) is beyond the parameter in our consideration 
of the representation, this feature of the conformal inversion wq is valid for 
any (711,77,2) (see Section [2.51 for a list of key properties of this element wq). 

In this book, we establish an analogous result to COl for G = 0{n\ + 
1,77,2 + 1), that is, we find the unitary operator tt{wq) on L 2 (C) for the 
conformal inversion wo. 

Then, here is our main result in this context. 

Main Theorem (see Theorem 15.1.11) . Let n\,n2 > 1 and n = n\ + 712 > 4 

is even. Then the unitary operator ir(wo) : L 2 (C) — > L 2 (C) takes the form: 

tt{w ) = 3"c, 

where Jc is the involutive unitary operator given in Section \l.B . That is, 

tt(w )u(x)= [ K(x,x')u(x')dfi(x'), u&L 2 (C), (1.7.2) 
Jc 

where the distribution kernel K(x,x') is given in Theorem 1 1.3. 1\ 

As obvious corollaries of the representation theoretic interpretation of 
3~c as above, we have: 

Corollary 1.7.1. (Plancherel and inversion formulas, see Corollaries 15.1.21 
and ECU 

l|3~C"|| L 2(C) = IMU 2 (C) for u G L 2 (C), 
Jc 1 = 3" c on L 2 (C). 
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The inversion formula "S c = 3~c7 implies that the following relation 



is reciprocal. Such an inversion formula is sometimes referred to as a recipro- 
cal formula (see Titchmarsh |71| , for this terminology in a general setting) . 

It is noteworthy that Corollary 11.7.11 is stated without any language of 
group theory. It would be an interesting problem to give a straightforward 
proof of Corollary 11.7.11 from the definition of Bessel distributions (|1.3.ip - 
p.3.3p without group theory. 

By 1VI2 and M4, ir is a non-highest weight, minimal representation if 
and only if ni,ri2 > 1 and n > 4. Therefore, we have discovered another 
mysterious phenomenon: 

Corollary 1.7.2. The kernel K(x,x') of the unitary inversion tt(wo) is 
a not locally integrable function if and only if ir is a non-highest weight, 
minimal representation. 

The highest weight module case {n\ = 1 or ri2 = 1) 

In the case ri2 = 1 (likewise m = 1), it splits into the direct sum of a 
highest weight module ir + and a lowest weight module 7r_ when restricted to 
the identity component Go = 50o( n i + 1,2) of G = 0{n\ + 1,2) according 
to the decomposition (|1.3.4p . Both tt + and 7r_ are minimal representations 



We note that G is the conformal group 0(n\ + 1,2) of the Minkowski 
space IR" 1 ' 1 , namely, the Euclidean space M ni+1 equipped with the flat 
Lorentz metric of signature (ni,l). In this case our representation tt has 
been studied also in physics. The minimal representation ir + may be in- 
terpreted as the symmetry of the solution space to the mass-zero spin-zero 
wave equation. The representation 7r + arises also on the Hilbert space of 
bound states of the Hydrogen atom. 

It is known that highest weight representations can be extended to holo- 
morphic semigroups of a complexified Lie group Go This theory has been 
initiated by Olshanski [58] and Stanton [66], among others, in connection 
with the Gelfand-Gindikin program to realize a family of representations in 
a geometrically unified manner. In this context the unitary operator ir(wo) 
may be regarded as the boundary value of a holomorphic semigroup. We 
then ask an explicit form of the holomorphic semigroup. This idea was 




of G . 
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first exploited by Howe [35] for the Weil representation, where he showed 

that w{wq) (= e z 3^) is given as the boundary value of the Hermite 
semigroup, i.e., the holomorphic semigroup with the Mehler kernel (see also 
Folland [18] for an exposition). The same idea also works in our setting of 
0{n\ + 1, 712 + 1) with ri2 = 1, and the explicit formula of a certain holomor- 
phic semigroup (the 'Laguerre semigroup') yields the formula of the unitary 
inversion operator tt(wo) by taking its boundary value in [441 146j . In this 
book, as a special case (i.e. ri2 = 1) of Theorem 15.1.14 we give a new proof 
of the formula of tt{wq). 

1.8 Uncertainty relation — inner products and fr- 
actions 

In this section, we consider the models of representations in the previous 
section in a more general setting, and formalize two representation theoretic 
questions (see Problem ll.8.1j) . 

Let P = LN = MAN be a parabolic subgroup of a real reductive Lie 
group G, P = LN its opposite parabolic subgroup, and g = n + I + n 
the corresponding Gelfand-Naimark decomposition of the Lie algebra g. 
Assume that the nilradical n is abelian, and in particular P is a maximal 
parabolic subgroup. 

Take a (non-unitary) one-dimensional representation \ '■ L ~^ C x , and 
consider the induced representation W := Indp(x®C). Then the space W°° 
consisting of its smooth vectors can be regarded as a subspace of (C°°r\$')(xi) 
by the restriction to n ~ iV C G/P. Here, S'(n) denotes the space of 
tempered distributions on n (regarded as the Euclidean space). Then, taking 
the Euclidean Fourier transform, we have 

W °° c §'(n) A S '(n), 

where n is identified with the dual space of n. We let G act on ^(W 00 ) 
through H. This G-action cannot be extended to S'(n), but its restriction to 
the parabolic subgroup P = LN can be extended to S'(n) because P acts on 
S'(n) just by translations and multiplications of unitary characters. Let wq 
be an element of K such that wqLwq 1 = L and wqNwq 1 = iV (a conformal 
inversion) . 

Now we consider the following setting: 

1) Let (-7T, "K) be an irreducible unitary representation of G, such that the 



2(3 



underlying (qc, -RT)-module 'Kk is a subrepresentation of W, and the 
Hilbert space Jf is realized in S'(n). 

2) Let G be an L-orbit in n, having an M-invariant measure dfj, such that 
L 2 (C, d/i) C §'(n). Further, P = LN leaves L 2 (C, dfi) invariant, and acts 
as a unitary representation. 

3) = L 2 (C,dfi). 

We note that the condition (2) determines the absolute value \x( a )\ f° r 
a G A. As we mentioned, the P-action on L 2 (C) is given just by translations 
and multiplications of unitary characters. Since G is generated by P and 
wo, the action it of the whole group G on L 2 (C, d[i) is determined by finding 
the formula of tt(wo). 

The (Euclidean) Fourier transform 3~ transfers the defining ideal for the 
affine variety C in n to the system, to be denoted by M, of differential 
equations on n such that the space "K is contained in the solution space: 

SoZ(M) := {/ G S'(n) :Pf = for any P G M}. 

In the previous example, n ~ M. n , C is the isotropic cone, and the system M 



is generated by Drii^ = 




Thus, we have two models of the irreducible unitary representation with 
the following nature: 

Solution model on SoZ(M) in S'(n). 

• The 'intrinsic inner product' on the Hilbert space "K (c SoZ(M)) is 
not clear. 

• The G-action on J£, to be denoted by w(g), is simple (essentially, 
the translations by the Mobius transform of G on G/P '= n). 



L 2 -model on L 2 (C,dfi) C S'(n). 

• The inner product on the Hilbert space L 2 (C, d/i) is very clear. 

• The G-action on L 2 (C,dii) is not simple except for the P-action. 



Now, we have a kind of 'uncertainty relation' in the sense that it is hard to 
find a single model having explicit descriptions of both G-actions and inner 
products. This feature in the above two models is symbolically summarized 
as follows: 
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SoZ(M) 


L 2 (C,dfi) 


inner product 


? 


simple 


G-action 


simple 


? 



Then, we ask: 
Problem 1.8.1. 

1) (conserved quantity for solution model) Find an intrinsic inner product 
on the solution space SoZ(M) that is invariant by G. 

2) (generalization of the Fourier-Hankel transform for L 2 -model) Find an 
explicit formula for the unitary inversion operator tt(wo). 

1) Solution model. In order to clarify the meaning of 'intrinsic inner 
product', we list three approaches to describe the inner product on 'K in 
SoZ(M). 

1-a) (Parseval type formula) Describe the inner product on "K according to 
the -ftT-type decomposition. 

1-b) (Green function) Give an integral expression of the solutions to M, and 
describe the inner product on "K by means of the integral expression. 

1-c) (Conserved quantities) Find an inner product formula in terms of only 
solutions. 

The approaches (1-a) and (1-b) give explicit inner products in a sense and 
are usually sufficient for representation theoretic purposes (e.g. showing the 
unitarizability) , however, do not give an intrinsic formula in the sense that 
the formula depends on the .fT-type decomposition or on the integral expres- 
sion of solutions. The approach (1-c) seeks for an intrinsic formula based 
purely on solutions. Here are a few remarks on (1-a)— (1-c) in order. 

The approach (1-a) is algebraic. We note that (G, L) forms a reduc- 
tive symmetric pair under the assumption that the nilpotent radical n is 
abelian. Consequently, (K, L n K) is a compact symmetric pair, and there- 
fore Indp(x®C) is K -multiplicity free. Then, the unitary inner product on 
"K is a scalar multiple of the L 2 -inner product on L 2 (K/L n K) on each K- 
type by Schur's lemma. Thus, the unitary inner product on "K is expressed 
by the 'weight function' 

m : K LnK -> R + , 
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where Ki n x = {t G K : Hom^ (r, L 2 ( K/L n K)) / 0} is explicitly known 
by the Cartan-Helgason theorem. The weight function m for unitarizable 
subquotients have been computed for a number of degenerate principal series 
representations, especially since the influential paper [36] by Howe and Tan. 
More generally, discretely decomposable branching laws for non-compact 
subgroups H give an extension of the approach (1-a) (see |48j ) . 

The approaches (1-b) and (1-c) are analytic. The integral expression of 
the solutions in (1-b) corresponds to the Knapp-Stein intertwining operator 
A in representation theory. The G-invariance of the resulting bilinear form 
/i;/2 l— ► (fi,Af2) is clear, and the positivity of the bilinear form implies 
the unitarizability of Image A This formula of the inner product is explicit, 
however, the formula is not written directly in terms of solutions. In fact, 
it is non-trivial to find a preimage / such that u = Af when a solution u is 
given. 

What we seek for in (1-c) is to describe directly the inner product on 
solutions. The energy for the wave equation is invariant under the time 
translation, and is a classic example of conserved quantities. The unitariz- 
ability of a solution space to M predicts the existence of positive definite 
conserved quantities. If the solution space is an irreducible G-module, then 
such conserved quantities must be unique up to scalar. This is what we call 
the 'intrinsic inner product' on the solution space. The uniqueness and the 
existence is predicted by representation theory. Finding its explicit form 
would be a challenging problem in analysis, arising naturally from represen- 
tation theory. 

2) L 2 -model. The existence of L 2 -models of small representations has 
been found for some other reductive groups. See [8] 1321 [74] for unitary high- 
est weight representations, [13] for spherical cases by using Jordan algebras; 
|62| for SO(4, 3), [73] for minimal representations of general reductive groups 
by using amalgamation of maximal parabolic subgroups. 

In the setting that we discussed as L 2 -model, there is a simple action of a 
parabolic subgroup P on L 2 (C), and we know the existence of the G-action 
on L 2 (C) by some other reasons. On the other hand, it is often the case 
that the missing piece is the explicit global formula on how to extend the 
action on L 2 (C) from P to the whole group G. 

Our subject stated in Problem 11.8.11 (2) is to fill this missing piece by 
finding the formula of the 'unitary inversion' tt(wo) for the conformal inver- 
sion element w$ £ G. We remark that the operator ir(wo) in the L 2 -model 
can be written as 

Tr( Wo ) = Jom(w ) or 1 , (1.8.1) 
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where w{wq) is a simple action on the solution model SoZ(M). However, the 
right-hand side of the formula (jl_._8. lT) does not give a solution to Problem 
11.8.11 (2) readily, because it is not easy to carry out the computation of the 
composition of integral forms in general. 

On the other hand, finding the formula of the unitary inversion tt(wq) in 
the L 2 model L 2 (C) has a significant meaning. To see this, we recall that 
there is a Bruhat decomposition G = P U PwqP. Therefore, once we get a 
formula of tt(wq), then the whole group action of G can be written by using 
7r(u>o) at most once (without any further composition of integral operators). 
Thus, the formula of tt(wo) is critical for finding the global formula of the 
whole action of G. 

We have seen in Sections ll .61 and [TTTI that Problem ll.8.11 (2) is settled for 
the Weil representation for the metaplectic group Mp(n, M) and the minimal 
representation of the indefinite orthogonal group 0{n\ + 1, 712 + 1), respec- 
tively. More generally, in the case that (tt, !K) is a minimal representation 
of a reductive group G, we expect that the operator could be described by 
means of some 'special function' of one variable. 

1.9 Special functions and minimal representations 

Yet another theme is special functions. 

In this book, we shall see special functions arise from the minimal rep- 
resentation. For example, _ftT-Bessel functions appear as the radial part 
of _RT-finite vectors in the Schrodinger model L 2 (C). Meijer's G-functions 
appear as the radial part of the integral kernel of 9^. Appell's hypergeo- 
metric functions bridge two models of the minimal representation, namely, 
the Schrodinger model and the conformal model. All together, we develop 
a new line of investigation on various special functions in connection with 
the minimal representation. 

The 'radial part' of the unitary inversion 

We begin with the (Euclidean) Fourier transform U^n as an illustrative 
example. As we already discussed, this corresponds to the unitary inversion 
operator for the (original) Schrodinger model of the Weil representation of 
Mp(n,R). 

According to the polar coordinate 

R+ x S 71 ' 1 ->■ M n , (r,w)^ra, 
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we have a unitary equivalence: 

L 2 (R n ) ~ L 2 {R + , r n - l dr) § L 2 ^™" 1 ) 

oo 

~ J]® L 2 (M+, r n - ! dr) <8) ^(M n ), (1.9.1) 
z=o 

where ft l (R n ) is the space of spherical harmonics of degree I, {(p £ C°°(S n ) : 
A Sn -iip = —l(l + n — 2)ip} (see Appendix l7.5p . Here, <8> stands for the Hilbert 

completion of the tensor product space, and > stands for the Hilbert 
completion of an algebraic direct sum. 

Correspondingly to the direct sum decomposition (|1.9.1j) . the Fourier 
transform 3~Rn is decomposed as 

oo 

9"r" = ^ Ti ® id . 

1=0 

Here, 7} is the Hankel transform of the following form (see Remark 14. 1.3j) : 

(T,/)(r) = 4 J n= 2 ± 2 L (x)f(-)dx. 
r Jo 2 \ r / 

Next, we consider our minimal representation of 0{n\ + 1, 712 + 1) realized 
on L 2 (C). Then, the bipolar coordinate on the isotropic cone C, 

R + x S" 1 - 1 x S n2 ~ l -> C 

induces a unitary equivalence 

L 2 (C) ~ L 2 (M+, ir n ~ 3 dr) § L 2 {S ni ~ l ) ® ^(S™ 2 " 1 ) 

oo 

~ ^® L 2 (M+, -r n ~ 3 dr) Jt'(R ni ) IK fe (R re2 ). 

Z,fc=0 

Then, the unitary inversion operator 3^c is decomposed as 

oo 

?c = Yl r ljfc ®id®id, 

Z,fc=0 

where T}^ is a unitary operator on L 2 (M_|_, |r n_3 dr). The unitary operators 
Ti k may be regarded as a generalization of Hankel transforms. 

It turns out that the kernel function of Tj ^ is real analytic (see Theorem 
1.9.1 below). This is a good contrast to the fact that the unitary operator 
S^C ° n -^ 2 (C) is given by a distribution kernel for general n\,ri2 > 1 (see 
Theorem 1.3.1). 
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Theorem 1.9.1 (see Theorem 14. 1 . 1 [) . Let Gq4 be Meijer's G-function (see 
Appendix 7.6 for definition), and we define a real analytic function by 



K hk {t) := 4(-l) 




l + k 



o + 2 



n\ — 712 + 4 — I — k 



6 + 2) 



2 



2 



where 



a : = max 



—ni — l+k —ri2+l—k 
2 ' 2 




:= mm 



—ni — l+k —n.2+l—k 
2 ' 2 



) 



Then, we have 



I POO 

m,kf)(r) = - / K ljk (rr')f(ry n - 3 dr' 
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It is noteworthy that Meijer's G-functions arise in the representation the- 
ory of reductive Lie groups. We observe the Casimir operator of a maximal 
compact subgroup K acts on L 2 (C) as a fourth order differential operator. 
Correspondingly, Meijer's G-functions G^(x\bi, &2, 63, 64) solve ordinary dif- 
ferential equations of order four (see (17.6.6P ): 



In the case n\ = 1 (or n 2 = 1), our minimal representation ir is a direct 
sum of a highest weight representation and a lowest weight representation. 
In this case, the kernels K\ & collapse to Bessel functions, and the unitary 
operators 7} ^ are reduced to Hankel transforms. 

The group law Wq = 1 in G implies 7r(u>o) 2 = id, and consequently, 
T 2 k = id for every l,k 6 N. Hence, Theorem 11.9.11 gives a group theoretic 
proof for the Plancherel and reciprocal formulas on Meijer's G-transforms 
which were first proved by C. Fox [19] by a completely different method. 

Corollary 1.9.2 (see Corollary 14.1 .40 . Let b\, 62, 7 be half-integers such that 
61 > 0, 7 > 1, ^-rp- <b2<\ + b\. Then, the integral transform 



is a unitary operator on L 2 (M + ). 

Corollary 1.9.3 (see Corollary I4.1.5p . The unitary operator Sb 1) b 2 a * s °f 
order two in L 2 {E. + ), that is, (Sf, lt t, 2 ,~/)~ l = <Sbi,b 2 ,7- 



.7=1 



561,62,7 : f( x ) ^ - 




u 1 

GH((xy)- I 61,62,1-7-61,1-7- b 2 )f(y)dy 
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A special case of the above corollaries (i.e. r&2 = 1 case) yields the classic 
formulas of the Hankel transform (see Remark I4.1.6P . 

A'-finite vectors in L 2 {C) 

By the general theory due to Vogan [76] , A- types of minimal representa- 
tions 7r are indexed by a natural number a = 0, 1, 2, . . . (by this property, ir 
is an example of the so-called ladder representation). In contrast to the con- 
formal model on §ol(A SP -i xSq -i) where explicit A-finite vectors are given 
readily by spherical harmonics, it is not clear a priori what A-finite vectors 
look like in the Schrodinger model L 2 {C) because the whole group A cannot 
act on the isotropic cone C. 

Our idea is to compute explicitly the intertwining integral operator T^ 1 
between these two models in Diagram 11.7.11 Then, by using a reduction 
formula of Appell's hyper geometric functions, we have (loosely): 

T" 1 (Gegenbauer polynomials) = A-Bessel functions 

and prove the following result: 

Theorem 1.9.4 (see Corollary I3.1.2p . Let n\ > n 2 . For a = 0, 1, 2, ... , 

no — 1 

r a ~^-Kn 2 - 1 (2r)4>{w) (0 G K a (R ni )) 

2 

is a A -finite vector in L 2 (C). In the K-type formula (see (|2.1.6jW . this 
vector belongs to the K-type Ji a (R ni+1 ) <g> ( K a+ ^— {W 1 ^ 1 ). 

In Theorem 11.9.41 the a = case corresponds to the minimal A-type, 
and was previously proved in [49 \ Theorem 5.8]. We note that ir is spherical 
if m = U2- Even in the case of spherical representations, finding explicit 
forms of the A-fixed vectors in L 2 -model is non-trivial. See [13] for similar 
formulas of the A-fixed vectors in L 2 -models for some other groups by means 
of A-Bessel functions. 

1.10 Organization of this book 

This book is organized as follows. We review quickly the L 2 -realization (a 
generalization of the classic Schrodinger model) of the minimal representa- 
tion of 0(p,q) in the first half of Chapter 2, and develop a basic theory of 
fundamental differential operators on the isotropic cone C in the latter half. 
Then we find some A-finite vectors on L 2 (C) explicitly by means of the 
A-Bessel function K v (z) in Chapter [3l Chapter [U is devoted entirely to the 



33 



integral formula of the unitary operator 7]^ on L 2 (M + , r p+q ~ 5 dr) for double 
spherical harmonics expansions (see Theorem I4.1.ip . In Chapter building 
on the results of Chapter [H we complete the proof of our main theorem (see 
Theorem 15. 1 . X [) . In order to make the proof readable as much as possible, 
we collect in Appendix the formulas and the properties of various special 
functions used in this book. 
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Chapter 2 



Two models of the minimal 
representation of 0(p, q) 



This chapter gives an account of the connection of the following two topics: 

1) Analysis on the isotropic cone C for commuting differential operators 
Pj (1 < j < n) associated to the quadratic form of signature (ni,rt2) (see 
Introduction OHO]) . 

2) Minimal representation of the indefinite orthogonal group 0(p,q). 
Throughout this chapter, we shall use the following notation: 



The first half of this chapter is a review from |47l [49] about two concrete 
models of the minimal representation of the group G = 0(p,q), namely, the 
conformal model (tu p ' q , V p ' q ) using the Yamabe operator (|2.1.ip in Section 
12.11 and the L 2 -model (the Schrodinger model) (ir , L 2 (C)) in Section |2"T21 
In the terminology of representation theory of reductive Lie groups (e.g. 
|41U78j ). the former realization gives a subrepresentation (the -fT-picture, the 
iV-picture, etc.) of a degenerate principal series representation, whereas the 
latter corresponds to the dual of the iV-picture via the (Euclidean) Fourier 
transform CF. 

The intertwining operator T between these two models will be given in 
(2.2.7), which is summarized as the following diagram: 



p = n\ + 1, q = n2 + l, n = n\ + n 2 = p + q - 2. 



L 2 (C) 
-ftT-picture 



T 



§'(RP+i- 2 ) 

T $ 
iV-picture. 
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Here, T is the identification map between functions on C and distributions 
supported on C by the canonical measure on C, is the G-intertwining 
operator between the iT-picture and the iV-picture, and is interpreted as the 
twisted pull-back for the conformal map ^ : ]R J,+<7 ~ 2 — ► S' p ~ 1 x S q ~ 1 . 

The latter half of this chapter is new. In Section 12.41 we analyze a com- 
muting family of differential operators Pj{b) (1 < j < n) of second order 
with parameter b in M n , and prove that they are tangential to the isotropic 
cone C if b = 1. The resulting differential operators Pj := Pj(l)\c on C^°(C) 
extend to self-adjoint operators on L 2 (C) (see Theorem I2.4.ip . Thus, we get 
a family of commuting differential operators Pj (1 < j < ra) of second order, 
which we call fundamental differential operators on C. In Section 12751 we see 
that the unitary inversion operator $c = ^(wo) diagonalizes Pj (1 < j < n) 
and that the intertwining relation of Pj and the multiplication by coordinate 
function Xj characterizes $c up to scalar. Thus, we develop an abstract the- 
ory of the unitary operator "Sq on L 2 (C) by taking the (Euclidean) Fourier 
transform S^n on L 2 (IR n ) as a prototype (see Theorems 12.5.21 f2.5.4|, [27575]) . 



2.1 Conformal model 

This section summarizes the conformal model of the minimal representation 
of the indefinite orthogonal group G = 0(p,q) (p + q : even). The advantage 
of the conformal model is that the group action on the representation is 
simple and that its geometric idea is clear. Since this conformal model 
corresponds to a subrepresentation of the most degenerate principal series 
representations (with a very special parameter), the same representation 
can be studied also by the purely algebraic method of (g, -?C)-modules. See 
[HI [361 ED] in this direction. The same subrepresentation can also be captured 
by the theta correspondence arising from the dual pair 0(p,q) ■ SL(2,R) C 
Sp(p + q,M) (see [80])- Our approach in this section is geometric, and the 
basic reference here is [37]. See also [43] for an elementary exposition from 
viewpoints of conformal transformation groups. 

The general geometric idea here is summarized as follows. Let X be 
an n-dimensional manifold equipped with a Riemannian (or more generally, 
pseudo-Riemannian) structure g. Then, associated to g, we define: 

k : the scalar curvature on X, 
Ax ■ the Laplace-Beltrami operator on X, 
n-2 

Ax ■= Ax + — ; rK (Yamabe operator). (2.1.1) 
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Then, although these objects depend on the (pseudo-)Riemannian structure 
g, the solution space 

§ol(A x ) := {f e C°°(X) : A x f = 0} 

is conformally invariant, namely, if (p : X — > X is a conformal diffeomor- 
phism with a conformal factor Cm G C°°(X) satisfying 

* 2 
¥> 9 = c^g 

then 

_ n — 2 

C-oopQ - C°°(X), /-c^ 2 /op (2.1.2) 

leaves SoZ(Ax) invariant, and hence we get a representation of the confor- 
mal transformation group Conf(X, g) on §ol(A x ) (see [471 Theorem A]). 
The point here is that the above construction is functional under conformal 
maps, and in particular, if two pseudo-Riemannian manifolds are confor- 
mally equivalent (not necessarily isometric), then the resulting two repre- 
sentations are isomorphic. 

A special case applied to pseudo-Riemannian manifolds which are con- 
formally equivalent to fiat pseudo-Riemannian space forms gives rise to the 
minimal representations of the indefinite orthogonal groups. Let us explain 
this specific case in more details. 

We denote by W' q the Euclidean space W +q equipped with the pseudo- 
Riemannian structure gup-q of signature (p,q): 

ds 2 = dx\ + • • • + dXp — dy\ — ■ ■ ■ — dy 2 . 

Then, the restriction of ds 2 to the submanifold 

M := {(x,y) G M. p+q : \x\ = \y\ = 1, x 6 R p , y G R q } (2.1.3) 
~ SP- 1 x S^ 1 

is non-degenerate, and defines a pseudo-Riemannian structure on M of sig- 
nature (p — l,q — 1). Here, | • | stands for the usual Euclidean norm. The 
resulting pseudo-Riemannian structure gu on M is nothing but the direct 
product of the standard unit sphere 5" p_1 (positive definite metric) and the 
unit sphere S 9-1 equipped with the negative definite metric ((— l)x the 
standard metric). 

Then, the Yamabe operator Am of M takes the following form (see [491 
(3,4,1)]): 

Ari = Agp-i — Agg-i — +( g ^) , (2-1.4) 
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where Agp-i and A^g-i are the Laplace-Beltrami operators on S p ~ and 
gq-i respectively. 

The indefinite orthogonal group G = 0(p,q) acts naturally on M p ' q as 
isometries. This action preserves the cone 

E:={(x,y)eM. p ' q :\x\ = \y\^0} 

but does not preserve M. In order to let G act on M, we set a function v 
on W> q by 

i/:RP»9->R, (x,y)^\ x \. 

If v G M(c 5) and h e G, then h ■ v *E E, and consequently 4^ G M. 
Thus, we can define the action of G on M: 

L h :M^M, v^J^- (h€G). 

Then, we have L* h gM = v ^. v yi 9m at T V M and thus the diffeomorphism is 
conformal with respect to the pseudo-Riemannian metric on M. Conversely, 
any conformal diffeomorphism of M is of the form for some h E G (see 
021 Chapter IV]). 

By the general theory (see (|2.1.2p ) of conformal geometry, we can con- 
struct a representation, denoted by w p,q , of G on the solution space to Am 
in C°°(M): 

V™ := §olA M = {/ G C°°(M) : A M f = 0}, 

where we set 

{w^{h- l )f){v) := v(h ■ v)-*^f{L h v), (2.1.5) 

for h £ G, v £ M, and / G y p,<? . The following theorem was proved in [55] 
in this geometric framework. There are also algebraic proofs (see Remark 
12X2) . 

Fact 2.1.1 (see |491 Theorem 3.6.1]). Lei p,q >2 and p + q > 6 be even. 

1) (irreducibility) (w p ' q ,V p ' q ) is an irreducible unitary representation of 

G. 

2) (unitarizability) There exists a G-invariant inner product ( , )m on 
V p,q . Such a G-invariant inner product is unique up to a scalar multiple, 
and we shall normalize it in (|2.1.7p . 

We write V p ' q for the Hilbert completion ofV p,q , and use the same letter 
w p ' q to denote the resulting irreducible unitary representation. 
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3) (K-type formula) Let K ~ 0(p) x 0(q) be a maximal compact subgroup 
of G. Then, the restriction of (w p ' q , V p ' q ) to K decomposes into irreducible 
representations of K as follows: 

oo 

K a (R p )®K b (R q ). (2.1.6) 

Here, l K a (M p ) denotes the irreducible representation of 0(p) on the space of 
spherical harmonics of degree a (see Section \77h^ . 

4) (Parseval-Plancherel formula) On each K-type 3i a (R p ) ® 3{ b (R q ) for 
(a, b) £ N 2 such that a + — 0j or equivalently, a + S^ 2 = b + the 
unitary inner product (-, -)m is of the form: 

(F,F) M = (a + ^)\\F\\l HM) . (2.1.7) 

Next, we consider the following injective map (see [MJ (2.8.2)]) by 
vp : RP +q 2 ► M, mrfz)- 1 ^), 
where for 2 = {z\ z") G l"" 1 © W' 1 we set 

Then, ^ is a conformal map such that ^*gM = T(z)~ 2 g^ P -i, q -i . According 
to [471 Definition 3.4], the twisted pull-back *&* of the conformal map ^ is 
a linear map 

: C°°(M) -» C°°(M P+ ^ 2 ) 

given by 

(i7)W:=rW-^(/o$)(4 (2.1.8) 
The image M+ of \1/ is roughly the half of M: 

M + := {u = (uq,v! ,v!' lUp+q-x) e M :u + u p+q ^i > 0}. 

We note that induces a conformal compactification of the flat space 
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where ~ Z2 denotes the equivalence relation in M = S p ~ x S 9 defined 
by u ~ —u. 

The inverse of ^ : R p+q ~ 2 ^ M + is given by 

* [U ,U ,U ,«p+g-l) = ^ J («,«). 

We note that is the ordinary stereographic projection of the sphere 
SP- 1 if q = 1. 

We write = (\I r ~ 1 ) for the twisted pull-back (in the sense of [471 

Definition 2.3]) of the conformal map <I'~ 1 : M+ -> R p+q ~ 2 , that is, 

is given by 

(•T 1 WW - (^ii^)-^— i ("I) ), (2.1.9) 

2 UO + V p+q -l \V J 

where u = %vq, 1/, v", v p+q -i) G M+, v ,Vp+<?-i El, t/e M p_1 , v" G 
In the group language (e.g. [H]), (\E'*)~ 1 is the standard intertwining oper- 
ator from the iV-picture to the if -picture. The map (^*) _1 will be applied 
also to other classes of functions. 

Remark 2.1.2. Our manifold M is a double cover of the generalized flag 
variety G/P max by a maximal parabolic subgroup P max (see (2.3.9)). Then, 
(w p ' q ,V p ' q ) is identified with a subrepresentation of the degenerate principal 
series representation induced from a certain one- dimensional representation 
of _P max . In this framework, Fact \2.1.1\ was proved by Kostant f5Ujj for 
p = q = 4 and by Binegar and Zierau Ji}/, for general p, q satisfying the 
condition that p, q > 2 and p + q > 4. Zhu and Huang 18 Of identified this 
subrepresentation with the local theta correspondence associated to the dual 
pair 0(p,q) x SX(2,R) in Sp(p + q,M.) (to be more precise, its metaplectic 
cover) and the trivial one- dimensional representation of SL(2,M). 

Remark 2.1.3. If p + q > 8, then vu p,q becomes a minimal representation 
in the sense of Definition 1 1 . <CT1 (see ft^). 

2.2 L 2 -model (the Schrodinger model) 

In Sections 12.21 and 12.3} we summarize the known results on the Schrodinger 
model of the minimal representation (tt, L 2 (C)) of G = 0(p,q). The basic 
reference is |49j . 
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A naive idea here is the following. Since VP : W 1,q 1 — ► M is a conformal 
map between two pseudo-Riemannian manifolds, we have 

#*(SoZ(A M )) C §ol(n RP -x, q -i). 

Since M is compact, it follows from the formula (2.1.8) of ^f* that 

y*(C°°(M)) C §'(R p+q - 2 ), 

where S'(IR p+9_2 ) denotes the space of tempered distributions. By taking 
the Euclidean Fourier transform 9 r KP + 9 -2, we get 

(^ M p +9 -2 o <j>*)(SoZ(A M )) C {u £ S'(R p+q ~ 2 ) : Suppu c C}. 

It was proved in [59] that the right-hand side is contained in L 2 (C) (and in 
fact a dense subspace of L 2 (C)). Let us fix some notation to formalize this 
fact. 

We define a quadratic form by 

Q(x) := x\ + • • • 4- x 2 _i — x 2 — ■ ■ ■ — Xp +q _ 2 , (2.2.1) 

which is the defining polynomial of the isotropic cone C in W +q ~ 2 . 

The substitution 5(Q) of Q into the Dirac delta function 5 of one variable 
defines a distribution on W +q ~ 2 \ {0}, which is represented as a measure, 
denoted by dfi on C. Alternatively, this measure is obtained as the volume 
form a\c where a is any (p + q — 3) form such that 

dQ A a = dx\ A dx% A • • • A dx p + q -2 

(see |24| Chapter III, Chapter 2]). An example of such a is given by 

(— l)i~ l €jXjdxi A • • • A dxj A • • • A dx n 



i=i 

In the bipolar coordinate: 



211x1 



K+ x 5 P ~ 2 x S q ~ 2 ^ C, (r, to, rj) ^ , (2.2.2) 
the distribution 5(Q) is given by 

(6(Q),<P) =\l°° I I ^{( r ^)v +q ^drdud V (2.2.3) 

2 Jo Jsp- 2 J si- 2 V rr ?y 
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for a test function ip on W +q ~ 2 . Here, doj and dn denote the standard 
measures on S p ~ 2 and S q ~ 2 , respectively. By this formula, we see that S(Q) 
extends to a Schwartz distribution on W +q ~ 2 of measure class if p + q > 4 
because r p+q ~ 5 dr is locally integrable, then. Obviously, we have 

supp 5(Q) =CU{0}. 

Let L 2 (C) = L 2 (C,d/j,) be the Hilbert space consisting of square integrable 
functions on C. Thus, for a function ip on C, the L 2 -norm of p is given by 

\W\\mc) = \ I™ I I W™,rri)\ 2 r p+<} - 5 drdu;dr,. (2.2.4) 
2 Jo Jsp- 2 J si- 2 

Correspondingly to the coordinates, we have an isomorphism of Hilbert 
spaces: 

L 2 (M+, K p+q ~ 5 dr) § L 2 (S p ~ 2 ) g L 2 (S q ~ 2 ) ~ L 2 {C). (2.2.5) 

Here, we employ the usual notation §> for the Hilbert completion of the 
tensor product. 

If p + q > 4, then u t— > u5(Q) defines a continuous, injective map from 
the Hilbert space L 2 (C) into the space S'(IR p+ ' ? 2 ) of tempered distributions 
on RP +q - 2 : 

T : L 2 {C) -» S / (M p+,? - 2 ), « i — ► u5(Q). (2.2.6) 

See @9l §3.4]. 

Now, we are ready to introduce a key map which will give an intertwining 
operator between the conformal model and the L 2 -model. 

7 := (^ MP+9 -2 o r)- 1 o T = (§*)- x o 3^ 9 _ 3 o T. (2.2.7) 

For u e Cg°(C), Tu G C°°(M + ). We extend to a function on M + U 
(-M+) by 

(Tu)(-w) = (i)6M + ). 

We recall from Fact 12.1.11 that the inner product on V p ' q is given by 
the formula ( 2.1.7|) . Then, the main ingredient of [491 Theorem 4.9] can be 
restated as: 

Fact 2.2.1. T extends to a unitary operator (up to scalar) from L 2 (C) onto 
VP' q . 
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Remark 2.2.2. The definition (|1.2.1i ) of the (Euclidean) Fourier transform 
adopted in this book involves the scalar multiplication by (2ir)~^. Ac- 
cordingly, the normalization of T is different from that of \49\j by a scalar 
multiplication. In our normalization, we have 

\\M\ 2 = l\n\l H c) (utL 2 (c)) 

as we shall see in (|3.1.11|) . 

Through the unitary operator T, we can transfer the unitary represen- 
tation (zu p ' q , V p ' q ) of G = 0(p, q) to a unitary representation, to be denoted 
by TT p ' q , on the Hilbert space L 2 (C) by 

^(g) : = T" 1 o w p ' q (g) o T, g € G. (2.2.8) 

Hereafter, we shall write tt for Tr p ' q for simplicity. Obviously, tt is irreducible 
because so is w p ' q (see Section 12. ip . We note that the unitary inner product 
of 7r is nothing but the L 2 -inner product of L 2 (C). Naming after the classic 
Schrodinger model L 2 (M n ) for the Weil representation of the metaplectic 
group (e.g. [H])) we shall say the resulting irreducible unitary representation 
(71", L 2 (C)) is the Schrodinger model for the minimal representation of G = 
0(p,q). 

In the philosophy of the orbit method due to Kirillov and Kostant, the 
Schrodinger model may be regarded as a geometric quantization of the min- 
imal nilpotent coadjoint orbit m j n in rj*, the dual of q = o(p,q). We note 
that the isotropic cone C is a Lagrangean variety of the symplectic manifold 

O min ■ 

So far, we have introduced two models of (m p,q ,V p < q ) and (tt, L 2 {C)) 
for the minimal representation of G. In the realization of V p < q , the K- 
structure is very clear to see, while on L 2 (C), its i^-structure is not clear a 
priori. Generalizing the idea in [19] where we found explicitly an L 2 -function 
(essentially, a X-Bessel function) on C belonging to the minimal X-type, 
we shall explicitly find a formula of If- finite vectors of L 2 {C) in Chapter 3. 
This computation is carried out by reducing the intertwining operator T to 
that of the Hankel transform of the if-Bessel functions. 

2.3 Lie algebra action on L 2 (C) 

We continue the review of an easy part of the Schrodinger model L 2 (C) of 
G = 0(p, q) from Section 12.21 We shall explain how the Lie algebra q = 
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o(p,q) acts on smooth vectors of L 2 (C). The action of a maximal parabolic 
subgroup P max on L 2 (C) will be also described. The main reference of this 
section is [33] §3]. 



Our notation here has the following relation with the notation in Intro- 
duction (Sections ll.lfll.3p : 

p = ni + 1, q = n 2 + 1, 
n = n\ + n 2 = p + q — 2. 

Let eo, • • • , Sp+q-i be the standard basis of IR p+,? , on which G — 0(p, q) 
(see (]1.7.ip ) acts naturally. First, we define subgroups M+ ax , M max , K, and 
K' of G as follows: 



mo 

M max 
M max 

K 

K' 



{g £ G : g ■ e = e , g- e p+(? _i 
M™ ax U m M™ x 
G(~)0(p + q) 

K n M™ ax 



e P+g -i} ^ 0(p- l,g- 1), 

~0(p-l,g-l)xZ 2 , 

~ 0(p) x O(g), 

~ Q{p- 1) x 0(g- 1). 



Then, is a maximal compact subgroup of G as we already used in 
Section [2?T] and K' is a maximal compact subgroup of M™ ax . Corresponding 
to the maximal compact subgroup K, the Cartan involution 9 of G is given 
by ^G?) = t 9~ j an d its differential (by the same notation) is given by #(X) = 
— *X in the matrix form. 

We note that the group M™ ax acts on the isotropic cone C in W +q ~ 2 
transitively, and leaves the measure d[i (see Section [2.2p invariant. 

Next we set 

e ..fl (1<J<P-1), 
* 3 ' \-l (p<j<P + q~2). 

Let Nj,Nj (1 < j < p + q — 2) and -ff be elements of the Lie algebra 
= o{p,q) given by 



(2.3.1) 



H 



E jfi ~ E j,P+q-l 
E 3,0 + E j,P+q-l 



a En 



c J^0j - ^j E p+q-l,j, 
e j E 0,j + e j E P+q-l,j, 



O.p+9-1 



We note that for 1 < j < p + q — 2, 



-2 e ,tf. 



(2.3.2) 
(2.3.3) 
(2.3.4) 



(2.3.5) 
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Then, we define abelian Lie algebras of q by 

p+q-2 
n max . = £ RNj, 

3=1 
p+q-2 

o := RH, 

and give coordinates for the corresponding abelian Lie subgroups jV max , 
iV max and A by 

p+q-2 

w+q -2 „ r ax ) a =( ai ,a 2 ,--- , a p+q - 2 ) ^ n a := exp( aj Nj), 

3=1 

p+q-2 

r p+?-2 „ ^55* o = (ai, 02, • • • , ap+q-2) » n a := exp( ^ a^-), 

3=1 

(2.3.6) 

E ~ A, t i — ^ exp(iff). 

Since M™ ax normalizes _/V max , we have a semidirect product group ^™ ax jv ma 
which has the following matrix form: 

M ma XArm ax = {g £ Q . ^ + = ^ + ^ ^ 

On the other hand, the natural action of G on M p+<? induces a transitive 
action on 

p— l p+g— l 

C:={(xo,-,^-i)eK P+9 \{0}:E^- E x ? = °>' ( 2 " 3 - 8 ) 

3=0 3=P 

with the isotropy subgroup j^fm ax jv max a ^ 6q _|_ e ^ +9 _ 1 Thus, we get a 
diffeomorphism 

G/M™ x iV max ~ C. 
We define a maximal parabolic subgroup by 

pSSx" - = jy/max^^yEix-^ (2.3.9) 
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Then the action of P max on L 2 (C) is described explicitly as follows ( \A9\ 
§3.3]): 

(n(m)rp)(x) = ip{\nx) (m G M™ ax ), (2.3.10) 

Km )V)(x) = (-1)^0*), (2-3.11) 
{-n{e tH )^)(x) =e- S± ^ ±t ^{e- t x) (tel), (2.3.12) 

(7T(fi a )^)(s) = e 2v /r l(«iXi+-H + ,- 2 x p+ ,_ 2 )^^ ( a G R P+9-2j_ 

(2.3.13) 



For g ^ P max , the action n(g) on L 2 (C) was not given in [49J. (In 
fact, the formula n(g) for general g G G is the main issue of the book.) 
Instead, we obtaind a formula for the differential action of the Lie algebra q 
in [32]. To state the formula, we recall some general terminology for infinite 
dimensional representations adapted to our special setting. 

Definition 2.3.1. Let tp G L 2 (C). We say ip is a differentiable vector if 

Ar W :=hm^M^ 
v ; t-*o t 

exists for any X G 0. Iterating this process, we say is a smooth vector if 
dir(Xi) ■ ■ ■ dir(Xk)tp is a differentiable vector for any k > 1 and Xi, . . . , X^ G 
g. (The notion of smooth vectors is defined for continuous representations 
on complete, locally convex topological vector spaces.) 
We say ip is K -finite if 

dime <C-span{7r(£;)'(/> : k G K} < oo. 

Let L 2 (C)°° be the space of smooth vectors of the unitary representation 
(tt,L 2 (C)) of G, and L 2 (C)k the space of K-finite vectors of L 2 (C). Ap- 
plying the general theory of representations of real reductive Lie groups (see 
[HIEE], f° r example) to our irreducible unitary representation (tt,L 2 (C)), 
we have 

1) L 2 (C)°° has naturally a Frechet topology, on which both the group G 
and the Lie algebra q act continuously. 

2) L 2 (C) K C L 2 (C)°° C L 2 (C). Moreover, L 2 (C) K is dense in L 2 (C)°°, 
and L 2 (C)°° is dense in L 2 (C), in each topology. 
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Prom now, we shall use the notation which is compatible with the nota- 
tion of Chapter [TJ 

n = ni + n 2 = (p — 1) + (q — 1). 

It follows from (2.3.12) and 

iV max on L 2 (C)°° is given as 

dir(H) -- 

Here, E := J27=l x *c5~ 1S ^ ne Luler operator, and Xj in (12.3.15P is the mul- 
tiplication operator by the coordinate function Xj. 

On the other hand, the differential action of _/V max on L 2 (C)°° is more 
complicated. In [491 Lemma 3.2], we gave its formula by means of second 
order differential operators in the ambient space W 1 = W +q - 2 via the inclu- 
sion map T : L 2 (C) ^ S'(M n ), u i-> u8(Q) (see {22M) as follows. Let Dj 
be a differential operator on W 1 (in the notation loc. cit., Dj = dw\ e (Nj) 
with A = p+ 2~ 4 ) given by: 

(n — 2 d n d d 1 n d 2 \ 

~ ^g^hox- + ek dxt) ) ■ 
3 fc=l K J k=l k 7 

(2.3.16) 

Then the differential action dn(Nj) (1 < j < n) is characterized by the 
commutative diagram: 

S'(M n ) 

Dj (2.3.17) 

§'{W l ). 

In Section \2A\ we shall treat these differential operators more systemat- 
ically by introducing the following differential operators Pj(b) (1 < j < n) 
with complex parameter b by 

Pj(b) := e^-D - (2£ + n - 26)—, (2.3.18) 
where we set □ = S^=i e j~j^?- definition, we have 




(1 < J < n). 



(|2.3.13p that the differential action of A and 



-(£ + 



n- 2 N 



2 v /r Tx i (1 < j < n). 



(2.3.14) 
(2.3.15) 



L 2 (C) 
L 2 (C) 
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In Theorem I2.4.H we shall see that the differential operator Pj := Pj(l) 
is tangential to C, and that the differential action dir(Nj) is given as 

diriNj) = ^ElejPjlc (1 < j < n). (2.3.19) 

2.4 Commuting differential operators on C 

In this section, we investigate basic properties of the differential operators 

d 

Pj(b) = ejXjD -(2E + n- 2b)— (1 < j < n) 

OXj 

on W 1 introduced in (2.3.18), and in particular, explain why and how the dif- 
ferential operators Pj(—1) (1 < j < n) (see f|2.3.18j) ) induce the differential 
operators -Pj(l) along the isotropic cone C. 
Again we recall 

n = n\ + n 2 = (p - 1) + (q - 1). 

We also recall from Section [TTT1 that R[x, ^] = . . . , x n , . . . , -^-} is 
the Weyl algebra and that ~R[x, J|] C ' is the subalgebra consisting of differ- 
ential operators tangential to C. 

The main result of this section is the following theorem concerning with 
the fundamental differential operators on the isotropic cone defined by 

Pj'.= Pj{l) (l<j<n). (2.4.1) 

Theorem 2.4.1. 1) PiPj = PjPi (1 < i,j < n). 

2) Pj G J^] c (l<j<n). 

3) The Lie algebra generated by 

[Pi,xj] = PiXj - xjPi (l<i,j<n) 

in the Weyl algebra R[x, ^] is isomorphic to o(p—l,q — l) +M, the Lie 
algebra of the differential action of the conformal linear transformation 
group CO(Q). 

4) Pj\c extends to a self-adjoint operator on L 2 (C). 

5) (E^ 2 )lc = «- 
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We shall give a proof of Theorem 12.4.11 in the following order: (2), (4), 
(5), (1), and (3). An important step for the proof is the following: 

Proposition 2.4.2. For u £ Cg° (R n \ {0}), we have: 

1) P,(-l)(u5(Q)) = (Pj(l)u)5(Q). 
Hence, we have Pj(—l)oT = To Pj(l). 

2) P j (l)(uQ) = (P j (-l)u)Q. 

In particular, Pj{l){uQ)\c = 0. 

Admitting Proposition I2.4.2"1 for a while, we give a proof of Theorem 2.4.1 
(2) and (4). 

Proof of Theorem \2. 4 -1\ (2). For a smooth function ip defined in an open 
subset V of C, we extend it to a smooth function ip in an open subset of 
M n \ {0}. Then, Pj{l)ip\y is independent of the choice of the extension 
and is determined by tp = ip\y ■ In fact, let tpi, ip2 be extensions of ip. Since 
{ip\—ip2)\c = 0) we find a smooth function locally defined in a neighborhood 
of V such that 

^1-^2 = uQ- 

Then (P^i - Pj^lc = (Pj(-l)u)Q\c = by Proposition [2X2] (2). There- 
fore, Pjtpi\c = Pjfp2\c- Thus, we have a well-defined map 

C °°(V) -+ C°°(V), t/i^Pjiltflv. 

Since this is a sheaf morphism, it is given by a differential operator on C. 
Hence, Pj = Pj(l) is tangential to C. Therefore, Theorem 12.4.11 (2) is 
shown. □ 

By Proposition 12.4.2] we also get the self-adjointness of Pj(l)\c as fol- 
lows. 

Proof of Theorem \2~4l\ (4). By ( 2.3.17|) . the differential action dir(Nj) is 
characterized by the relation 

Dj oT = T odir(Nj). 

On the other hand, we have proved in Proposition 12.4.21 (1) 

P j (-l)oT = To(P j (l)\ c ). 
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Since Dj = ^-^-€jPj(—l) by definition, we get 

MNj) = ^ ej p,(i)|c. 

As (tt,L 2 (C)) is a unitary representation of the Lie group G = 0(p,q), the 
differential action \/— ldir(X) on L 2 (C)°° extends to a self-adjoint operator 
on L 2 (C) for any X e g = o(p,q). Hence, Pj = Pj(l) is self-adjoint. Thus, 
Theorem 12.4.11 (4) is proved. □ 

Before giving a proof of Proposition 12.4.21 we se t U P the notation of 
the meromorphic continuation of the distribution Q(x)h_. For a complex 
parameter A with Re A > — 1 , we define a distribution on M n by 



Q(x) 




if Q(x) > 0, 
if Q{x) < 0. 



Then, Q(x)+ continues as a distribution depending meromorphically on the 
parameter A £ C (see Appendix 7.1). In particular, as a distribution on 
M. n \ {0}, Q{x)\ has only simple poles located at A = —1, —2, —3, .... Since 
the gamma function r(A + l) has simple poles exactly at the same places A = 

— 1, —2, . . . , we see that r(A+i) defines a distribution on R n \ {0} depending 
holomorphically on A € C. 

In view of the residue formula (j7.1.1|) in Appendix l7.1l for the one variable 

t x 

case r(A+i) ' we h- aYe following formula of generalized functions on W 1 \ 
{0} (see [Ml Chapter III, §2.2]): 

S(Q(x)) 



r(A + i) 



Therefore, the map T : L 2 {C) -> S'(R n ) defined in QfiTZM has the following 
expression: 

for u G C$°(R n \ {0}). The proof of Proposition l2"X2l will make use of 
(|2.4.2|) . Along this line, we prepare: 

Lemma 2.4.3. For u £ C^°(IR n \ {0}), 

□ = (Dti)Q+ + 4A(£u)Q^ 1 + 2A(2A + n - 2)uQ > T 1 . 
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Similar formulas also hold if we replace Q+ by Q for positive integers A. 
In particular, letting A = 1, we have 

U{uQ) = {Uu)Q + (4E + 2n)u. 

Proof. By the Leibniz rule, we have 



d 2 , \ \ d 2 u „ A _ du ^ } 



+ 2Ae i uQ^" 1 + 4A(A - l)^uQ^- 2 . (2.4.4) 
Summing up (|2.4.4h multiplied by the signature €j over j, we get Lemma. □ 
Here is a key formula for the proof of Proposition 12.4.21 

Lemma 2.4.4. For u G C^°(M n \ {0}), w;e /iawe the following identity as 
distributions on W 1 \ {0}: 

Pj(b)(uQl) = (Pj{b - 2X)u)Ql - 4A(A - fyejXjuQ^ 1 (2.4.5) 

for any 1 < j < n, b G C, and A £ C \ { — 1, —2, . . . }. 

Proo/. By (pQ~3|) . we have 

(2£ + n-26)^-(uQ^) = ((2£ + n - 2(6 - 2A)) J^-) 

+ 2A(4A + n - 2 - 26)e j x i nQ^ 1 
+ 4Ae i x j (£u)Q+~ 1 . 

Then, (j2~X5D follows from Lemma EES □ 

Now we are ready to prove Proposition 2.4.2, 

Proof of Proposition Since p(A+i) * s a distribution on W 1 \ {0} with 
parameter holomorphically dependent on A G C, we have 

for all A G C. By letting b = A, we have 



r(A + i)y v JV y T(A + i) 
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Further, by letting A = — 1, we get 

Pji-l^uSiQj) = (Pj(l)u)5(Q). 
By letting A = 1, we get 

Pj{l){uQ) = {P 3 (-l)u)Q. 
Therefore, we have proved Proposition 12.4.21 □ 

Thus, the proof of Theorem 2.4.1 (2) is completed. Next, let us prove 
the following: 

Proposition 2.4.5. ^ e^P^l) 2 = QU 2 . 
i=i 

Admitting Proposition 12.4.51 for the time being, we give a proof of The- 
orem EI (5). 

Proof of Theorem \2.4-l\ (5). We have already shown that Pj(l) £ M[x, — 



dxi 

(1 < j < n). Then, Theorem 12.4.11 (5) is an immediate consequence of the 
following identity in the algebra R[a;, ^ 

To prove Proposition 2.4.5, we list some basic relations of the Lie bracket 
[A, B) = AB - BA in the Weyl algebra R[x, J^]: 

Claim 2.4.6. 

1) [E,Xj] = Xj. 

^ ^ ' dxj ^ dxj 
3) [E,D] = -2D. 





4) [n,x j ] = 2e j 



dxj 



Here, xj denotes the multiplication operator by xj. 

Proof. Straightforward by the Leibniz rule. □ 
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Proof of Proposition \2.4-5\ In light of the definition 

Pi (b) = e iXi U -(2E + n-2b)-^-, 

we develop Pi(b)Pj(b) as 

Pi{b)Pj{b) = (I) + (II) + (III) + (IV), 
where (I) = (I)y(6) is given by 
(I) := (eiXia)(e jXj a) 



d 

and similarly, 



(II) := - (etxtD) (J2E + n - 2b)-^- 

d 

= -ei(2E + n-2b + 2)xi—n, 

OXj 

(III) := - ({2E + n- 2b)— ■) (e jXj D) 
= -e j (2E + n-2b)(x j — + 5 ij )u, 

(IV) := [{2E + n- ^)—^ ((2E + n - 2b) — 

d 2 

= (4E 2 + 4(n + 1 - 2b)E + (n-2b)(n + 2- 2b)) 



dxidxj 



Here, Sij stands for Kronecker's delta. Now, we take i = j and 6 = 1, and 
sum up these terms over j: 



£) 6^(1)^(1) = qd 2 + 2£d, 

3=1 

n 

£>(I%(1) =-(2E + n)ED, 

3=1 

£^(111)^(1) = -(2£ + n - 2)(£ + n) □, 

3=1 

n 

£3(^)^(1) = (4£ 2 + 4(n - 1)£ + (n - 2)n) □. 

3=1 
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Summing up these four equations, we get Proposition 12.4.51 □ 

It is easy to see that the formulas for (I), (II), (III), and (IV) used in the 
previous proof lead us also to the following: 

Proposition 2.4.7. As differential operators on W 1 , we have 

[P i (b),P j (b)]=0 

for any 1 < i,j < n and b € C. 

As a special case of Proposition 12.4.71 we have: 

Proof of Theorem \2. 4 -1\ (1). This follows from Proposition 12.4.71 by letting 
6 = 1. □ 

Finally, let us prove Theorem 12.4.11 (3) . 



Proof of Theorem 2.4-l\ (3). We continue the notation (|2.3.1R for ej = ±1 
(1 < j < n =p + q- 2). 

Let Eij (1 < i,j < n) be the matrix unit. Then 

' i< jl'ij ~ Eji (1 < i < j < n) 

forms a basis of the Lie algebra o{p —1,^—1) of the Lie group M™ ax ~ 

0(p — l,q — 1). Correspondingly, the natural differential action on M. n = 
M ( P -i)+(g-i) giveg a vector 

9 d 

Xij := titj^Q^ - Xj-Q^ (l<Kj< n). 
Likewise, the differential of the dilation is given by the Euler vector field 

Hence, Theorem 12.4.11 (3) is an immediate consequence of the following 
lemma. □ 

Lemma 2.4.8. As differential operators on M n , we have 

1) [P i ,x j ]=X ij (l<i<j<n). 

2) [Pi,xi] = -2{E + n- 2) (l<i<n). 
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Proof. By the definition (|2.3.18|) of Pj = Pj(l), we have 



r 

Yn--l) 
Ft 

[2E + n- 2, X j] 



- dxi ' ° 



[Pi,Xj] = €iXi[n,Xj] - (2E + n-2) 

d_ 

1 dxi ' 



By Claim I2T4.61 we have 



d d 
= 2ei€jXi— - 5ij(2E + n - 2) - 2x j-Q^r 

= 2Xij - 5ij(2E + n- 2). 

Hence, Lemma is proved. □ 

Hence, the proof of Theorem 2.4.1 is completed. 

Remark 2.4.9. Let R be the subalgebra of M[x, -§^] C generated by Xk, E, 
and Xij (1 < k < n, 1 < i < j < n). Then, Pj G R[x, J^f but Pj R. 

To see this, we say an element P of the Weyl algebra M.[x, ^] is of degree 

k if 

[E,P]= kP. 

If P,Q are of degree k, I, respectively, then PQ is of degree k + I because 

[E,PQ] = [E,P]Q + P[E,Q]. 

Since x^ is of degree 1, and E and is of degree 0, any element of R 
is expressed as a linear combination of operators of non-negative degrees. 
Since Pj is of degree —1, we conclude Pj ^ R. 

Remark 2.4.10. Our concern here is with the case n > 2 (i.e. p + q > 4). 
Let us examine the degenerate case where n = 2 and n\ = ni = 1 (i.e. 
p = q = 2). Then, the polynomial Q(x) = x\ — x\ is not irreducible, and the 
differential operators P\ and P2 have the following formulas: 

P 1 + P 2 = -(x l + x 2 )(— + — ) , 

By the change of variables 

yi = xi + x 2 , y2 = xi-x 2 , 
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the isotropic cone C is given by 

{(yum) e» 2 \M = vi = o or y 2 = o}, 

and we have 



P 1 + P 2 = -4 yi - o 



dy\ 

d 2 

P 1 -P 2 = -4y 2 -^. 



2.5 The unitary inversion operator "Jc = tt(wo) 

In this section, we list some important features of the element 

"* = (o -/,) e G - 

and investigate key properties of the corresponding unitary operator 

3~ c := k(wq) 

on L 2 {C). 

10 {Order two) Obviously, the element wq is of order two. Therefore, 3^ = 
id on L 2 (C). 

11 (Cartan involution) The Cartan involution 9(g) = is given as the 
conjugation by wo: 

6(g) = wogwv 1 (2.5.1) 
because fytOoS = wq for g £ G. 

12 [Center of K) wq lies in the center of K. This is obvious from (|2.5.ip . 
It also follows directly from the definition of wo in the matrix form. 

13 (Bruhat decomposition) Retain the notation as in Section [2.31 Then, 

Ad(w )H = -H, (2.5.2) 

and therefore Ad(u>o)|a = —id. We also see readily from (I2.3.5P and 
([237L]) that 

Ad(wo)Nj = ejNj (l<j<p + q-2), (2.5.3) 
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and therefore 

Ad(w )n max = n 5 ^. (2.5.4) 
The Gelfand-Naimark decomposition 

= ( m max + a + r? 5 ^) + n max = p^ + Ad(w )x^ 

gives, in turn, the following Bruhat decomposition of G on the group 
level: 

Q _ pi535" u , pS3x" JJ pSSx". (2.5.5) 

14 [Restricted root system) Let g = t + p be the Cartan decomposition, and 
we take a maximal abelian subalgebra b of p. Since Ad(wo) acts on p 
as — id, wo acts on the restricted root system E(g, b) as — id. We note 
that the longest element in the Weyl group W(S(g, b)) is equal to — id 
if p ^ q or if p = q £ 2Z. In the case p = g G 2Z + 1, the restricted root 
system E(g, b) is of type D q and 

- id £ W(E(g, b)) ~ & q x (Z/2Z) 9 " 1 , 

and therefore Ad(u>o)|b = — id gives an outer automorphism on E(g, b). 

15 {Jordan algebras) We regard M^ -1 ' 9-1 ~ W +q ~ 2 as the semisimple Jor- 
dan algebra with indefinite quadratic form Q(x) (see (jl.l.ip ). This Jor- 
dan algebra is euclidean if min(p, q) = 2, and non-euclidean if p, q > 2. 
The conformal group (Kantor-Koecher-Tits group) of W~ 1 >9 — 1 is equal 
to the group G = 0(p, q), and the action of the element wq on M. p ~ 1,q ~ 1 
corresponds to the conformal inversion x \— ► —x~ l (see |16|. I65|). Thus, 
we call wq the conformal inversion element. The structure group 

L + := M™ ax A ~ CO(Q) = 0(p - 1, q - 1) X R >0 (2.5.6) 

acts on M p_ 1,9-1 by x h-> e*mx for (m, e* H ) G M™ ax x ^4, and on its dual 
space by x i— > e _i t m~ 1 x. 

16 (T/ie action on the minimal representation) In the conformal model 
(w p,q ,V p ' q ) (see Section |2T|) . the whole group G acts on the geometry, 
and therefore, the unitary operator w p,q (wo) is easy to describe: 

(w p ' q (w )h)(v , . . .,v p+q -i) = h(v , . . . ,v p -i, -v p ,.. . ,-v p+q -i), 

(see the definition (I2.1.5P ). Then, by using the intertwining operator 
7 : L 2 (C) -> (see ff^T277|) ). we get the formula of "S c = n(w ) in the 
Schrodinger model (ir,L 2 (C)) as 

J c = T -1 o w p ' q (w ) o T. (2.5.7) 



57 



However, we do not know how to find the formulas (|1.3.1|) — ( |1.3.3|) of "Sq 
directly from (2.5.7) and the definition of T. Thus, in order to find the 
unitary inversion formulas of &c = ^(uJq), we shall take a roundabout 
course (by using the if -type decomposition in part) in later chapters. 

Now, let us study basic properties of the unitary inversion operator 3~c = 
tt(wo) on L 2 {C). First, we extend 5Fc to distribution vectors. 

Let L 2 (C)~°° be the continuous dual of the Frechet space L 2 (C)°° (Def- 
inition l2.3.ip . namely, the space of continuous C-linear maps L°°(C) — ► C. 
Elements in L 2 (C)~°° are called distribution vectors. 

We introduce a conjugate linear map 

t : L 2 {C) -» L 2 (Cy°° 

characterized by 

(v, t(u)) := / v(x)u(x)dfi(x) for any v G L°°{C). (2.5.8) 
Jc 

The inclusion 

L 2 (C)°° C L 2 (C) C L 2 (C)~°° 

is sometimes referred to as the Gelfand triple. 

From the standard theory of Sobolev spaces and the definition of smooth 
vectors, we have the following inclusive relation: 

C£°(C) C L 2 (C)°° C C°°(C). 

(To see L 2 (C)°° C C°°(C), it is enough to use the P max -action on L 2 {C).) 
Then, taking their duals, we get 

£'(C) C L 2 {C)-°° C D'(C), (2.5.9) 

where D'(C) stands for the space of distributions on C, and £'(C) for that 
of compactly supported distributions on C. 

For g G G, we extend the unitary operator 5 = n(g) on L 2 (C) to a 
continuous operator S on L 2 (C)~°° by 

(u,5u) := (^ 1 u,n) for u G L 2 (C)~°° and f G L 2 (C)°°. (2.5.10) 

Here, we have used the fact that the unitary operator S~ 1 on L 2 {C) induces 
a continuous map (we use the same letter 5 _1 ): 

S- 1 : L 2 {C)°° ^ L 2 {C)°°. 
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Then, the extension S satisfies 

S o l = i o S on L 2 (C), 

because for u G L 2 {C) and v G L°°(C) we have 

(v, Si(u)) = (S^ 1 v, t(u)) 

= (S~ 1 v,u) L 2( C ) 
= (v,Su) L 2 {C) 
= (v,l(Su)}. 

Likewise, the Lie algebra q acts on L 2 (C) _ °° by 

(M7f(.X» := -(d7r(A>,u> for u G L 2 (C)~°° and v G L 2 (C)°°. 

(2.5.11) 

Then, it readily follows that 

dn(Ad(g)X) = n(g) dv{X) tt^ 1 ) (2.5.12) 

for g G G and X G 0. 

We shall write simply 5" for 5, and dir(X) for d7r(X) if there is no con- 
fusion. 

For £ G C, we denote by 5^ the Dirac delta function at £ G C, namely, 

(v,8t) := v(£) 
for a continuous function v on C. Since 

: L 2 (C)°° -> C 

is a continuous map, we may regard <5g G L 2 (C)~°°. Then, we have 

J^k G ^ 2 (C)"°°. (2.5.13) 
Applying the definition (|2.5.10p to S = S^ 1 , we have for any v G L°°(C), 

= {?cv)(0. (2.5.14) 

Remark 2.5.1. In contrast to the (Euclidean) Fourier transform Ji", S^ 1 ^ 
is a real valued distribution. In other words, the kernel k( 

V y (2tt)^ 
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for Str" is not real valued, while the kernel K(x, £) for S~c below is real val- 
ued. This reflects the fact that both kernels k(x,£) and K(x,£) are charac- 
terized by the system of differential equations up to scalar (see Introduction 

roj) .- 

Pj k(-,0 =Zjk(;0 mR n (l<j<n), 
P j K(;£)=4€ j K(;0 inC (l<j<n), 

where pj = — \J — lgf-- A distinguishing feature here is that the self-adjoint 
differential operator pj of first order satisfies 

Pj = -Pj> 

while the self-adjoint differential operator Pj of second order satisfies 

For a fixed £ G C, J^ 1 ^ is a distribution on C by (|2.5.13|) and ([2X9]) . 
We shall see that this distribution is locally integrable on the open dense 
subset 

Thus, it is convenient to write the distribution 3^ 5^ as a generalized func- 
tion in the sense of Gelfand-Shilov [23] by using the canonical measure 
dn(x) = 5{Q{x)) on C, that is, 

for some generalized function K(-,£) on C. Then, the formula (2.5.14) can 
be expressed as 

(Sc«)(0= / K(x,Z)v(x)dn(x). (2.5.15) 
Jc 

The argument so far assures that K(-,£) is a generalized function on C 
for each fixed £ G C. On the other hand, since the sesqui-linear map 

L 2 (C) x L\C) - C, («, u) ' ► (3^, n) L2(c) 

is continuous, we may regard K(x,£) is a generalized function (or equiva- 
lently, K(x,£)d/j,(x)d/j,(£) is a distribution) on the direct product manifold 
C x C such that 

(? c v,u) LHc) = I v(x)^)K(x,Odfi(x)dfi(0 (2.5.16) 
JCxC 
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for any u, v G L 2 (C)°°. 

Now, we are ready to state basic properties of the 'unitary inversion 
operator' 3^c = ^(wo) as the counterpart of the properties |Ql |3] of the 
conformal inversion wo 6 G: 

Theorem 2.5.2. 

1) GFc is an involutive unitary operator on L 2 (C). In particular, we have 



K(x,Z)=K(£,x) (2.5.17) 
as distributions on C x C . 

2) Ifve L 2 (C)°°, then 3 c v G L 2 (C)°°. In particular, 

? c v G C°°{C) DL 2 (C) 
for any compactly supported C°° function v on C . 

3) For 1 < j < n, the following identities of operators on L 2 (C)°° hold: 

$c ° 4xj = Pj o 3~ c , 

J c ° Pj = Axj o g- c , 

3- c0j E = _(£ + n- 2) o$ c . 
These identities hold also on L 2 (C)~°°. 

4) For any t> G L 2 (C)°°, 

(Scr«)(0 = / v(x)K{x,S)dn{x). 
Jc 

5) For each fixed £ = (£i, . . . , £ n ) G C, i/ie generalized function K(£, •) 
solves the system of differential equations on C : 

P j K(S i -) = 4S j K{£ r ) (l<j<n). 

In turn, K(-,£) satisfies 

P j K(;0=^ j K(;0 (l<j<n). 

Remark 2.5.3. We shall see in Theorem 1 5. that 
as distributions on C x C . 
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Proof of Theorem 1 2. 5. 21 1) The first statement is straightforward from def- 
inition 3~<7 = tt(^o) because (it, L 2 (C)) is a unitary representation of G (Fact 
12.2. 1 p and Wq = 1. The second statement follows from (2.5.16) and 

(3c«,«) = (ff'cw,^) = (v.Fcu). 

2) The group G preserves L 2 (C)°°, and in particular, we have 

J C (L 2 (C)°°) = L 2 (C)°°. 

Since Cg°(C) C L 2 (C)°° C C°°(C) nL 2 (C), we get J C (C^(C)) C C°°(C)n 
L 2 (C). 

3) By (|2.5.3p in ][3l we have the following identities on L 2 (C)°°, and also 
on L 2 (C)-°° by (|23J2|) : 

J c o d7r(Fj) = ejdTriNj) o J c , 
y c o d7r(iVj) = €jd7r(Nj) o J c , 
? c o d7r(i?) = -dvr(if) o ? c . 

Now we recall the formulas of the differential action: 

dTr(Nj) = 2y/^l Xj (see (|2.3.15p ). 

d7r(jy,-) = ^-tjPj (see 

n — 2 

dir(H) = -{E+ — — ) (see (|2.3.l4|> ). 

These operators are defined on L 2 (C)°° C C°°(C), and in turn, they are 
defined on L 2 (C)"°° C D'(C) (see (12X91) ). Therefore, the third statement 
follows. 

4) Since Wq 1 = wq, we have [F^, 1 = &c, and therefore the statement is 
equivalent to what we have seen in (|2.5.15p . 

5) It follows from the third and fourth statements that we have (by 
switching x and £) 

/ ^KfoxWddfi® = (Pj) x [ K($,x)v(t)dfi(S) 
Jc Jc 

= [ (P^K&xMOdm 
Jc 

for any v E L 2 (C)°°. Hence, we have shown 

4$ j K($,x) = (P j ) x K{£,x). 

The second assertion follows from Pj = Pj as we saw in Remark l2.5.11 Thus, 
Theorem 12.5.21 has been proved. □ 
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If A is a continuous operator on L 2 (C), then A induces a linear map (we 
use the same letter) A : L 2 (C)°° — ► L 2 (C)°° on the space of smooth vectors. 

The next result shows that the intertwining relation in Theorem 2.5.2 
(3) characterizes the operator $c up to scalar: 

Theorem 2.5.4. Suppose A is a continuous operator on L 2 (C) such that 

A o Axj = Pj o A, 
AoPj = ixj o A, 

on L 2 (C)°° for 1 < j < n. Then, A is a scalar multiple of 3*c- In particular, 
A 2 is a scalar multiple of the identity operator. 

Proof. We set B := J^ 1 o A. Then, B satisfies 

B o 4xj = Axj o B, 
BoPj =p j0 B, 

on L 2 (C)°°, and consequently, 

Bodn(Nj) =d7r(Nj)oB, 
Bodir(Nj) = d-K(Nj)oB, 

for 1 < j < n. Since the vector space n max + H ssr = YTj=\ MiV j + E"=i 
generates the whole Lie algebra q, we have 

B o dTr(X) = dir(X) o B (2.5.18) 

for any X G g. In particular, such an operator B preserves the space of 
infinite vectors, namely, 

B{L 2 {C) K ) C L 2 (C) K . 

Therefore, the identities (|2.5.18p hold also on L 2 (C)k- 

On the other hand, it follows from Fact 12.1.11 and Fact 12.2.11 that G 
acts irreducibly on L 2 (C). Then, L 2 (C)k is an irreducible (g, X)-module. 
Therefore, B must be a scalar multiple of the identity by Schur's lemma. 
Hence, A is a scalar multiple of $c- The last assertion is an immediate 
consequence of the fact that 3^ = id (see 10). □ 

Theorem 2.5.5. For a fixed £ = (£i, . . . ,£ n ) G W 1 \ {0}, we consider the 
system of differential equations: 

Pj^ = A^ (l<i<n), (2.5.19) 

as distributions on C . 
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1) IfQ(£) / 0, then ip = 0. 

2) If Q(0 = 0, then the solution space in L 2 (C)~°° is one- dimensional. 
Proof. 1) It follows from (12.5.191) that 

n 

(£ / e j lf)il> = 4Q(ZW. 

3=1 

Then, by Theorem 12.4.11 (5), the left-hand side vanishes. Hence, ip = 
unless Q(C) = 0. 

2) Suppose £ € C. Taking the Fourier transform S'c of the differential 
equation (2.5.19), we have the following equation 

4^(3^) = 4^(3^) in L 2 (C)-°° 

from Theorem 12.5.21 (3). Hence, 

(x j -Z j )(2cil>) = (l<j<n). (2.5.20) 

Since 3 C ?P G L 2 (C)-°° C D'(C), the equations (|2.5.20) hold as distributions 
on C. Then, considering the equations (2.5.20) in a local chart, we see that 
Jcip is a scalar multiple of the delta function 5^. Conversely, 5^ solves 
(I2.5.20jl and J c G L 2 (C)-°°. Therefore, 9^ G L 2 (C)-°° solves (|2.5.19|) . 
Now, Theorem is proved. □ 
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Chapter 3 



7^-finite eigenvectors in the 
Schrodinger mo del L 2 (C) 

In the conformal model (see Section (2711 . we can find readily explicit in- 
finite vectors. However, it is far from being obvious to find explicit forms 
of if-finite vectors in the L 2 -model for the minimal representation. In this 
chapter, generalizing the idea of [491 Theorem 5.8] for the minimal K-type, 
we find explicit vectors in L 2 (C) for every K-type, by carrying out the com- 
putation involving the integral operator 7 : L 2 (C) V p ' g (see Fact I2.27T]) . 

3.1 Result of this chapter 

Throughout this chapter, we assume p > q > 2 and p+q > 6. For (I, k) £ N 2 , 
we consider the following two (non-exclusive) cases: 

Case 1 : + I - k > 0, 

2 " (3.1.1) 

Case 2 : ^ + / - k < 0. 

The case 2^ + / — k = belongs to both Cases 1 and 2. This overlap will be 
convenient later because all the formulas below are the same for both Cases 
1 and 2 if (I, k) satisfies ^ +l-k = 0. 
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For (I, k) EN, we define real analytic functions fi^ k on M + by 



fi,k( r ) :z 



.9-3 



p-3 



+l K^ +k {2r) Case 1, 



r 



2" +fc K Ell 3 +i (2r) Case 2, 



(3.1.2) 



. K q -3 A2r) Case 1, 
#j»=3 +l (2r) Case 2. 
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Here, K u (z) is the .ff-Bessel function, i.e., the modified Bessel function of the 
second kind (see Appendix \7.2\i and K v {z) = (§) _l 'K v {z) is the normalized 
if-Bessel function (see (f7T6|) l. 

By using the polar coordinate (2.2.2), we define a linear subspace Hi ^ 
of C°°(C) consisting of linear combinations of the following functions: 

fi, k (rm^(v) (^eiK'Or- 1 ), ^e^- 1 )). (3.1.4) 

Here, !K J (M m ) denotes the space of spherical harmonics of degree j (see 
Appendix I7.5p . 

We recall from Section [2]3] that there are two key compact subgroups for 
the analysis on the minimal representation L 2 (C): 

K ~ 0(p) X 0{q), 

K' = K n M™ ax ~ 0(p - 1) x 0(q - 1). 

We note that the i^'-action on L 2 {C) is just the pull-back of the .ff '-action 
on C (see (|2.3.10p ). but the if-action on L 2 (C) is more complicated because 
K cannot act on C. Then, here is our main result of this chapter: 

Theorem 3.1.1. For each pair (I, k) G N 2 ; we have 

1) (asymptotic behavior) Hi ^ C L 2 (C) for any l,k £ N. 

2) (K-type and K' -type) H hk ~ ^(RP- 1 ) <g> ^(M"" 1 ) as a K' -module. 

Furthermore, H^ k belongs to the K-type Ji< l ' k \W) <g> JC a (W +£ ? (Rff) of 
L 2 (C). Here, we define a non-negative integer a(l, k) by 

a(L k) := max(Z, k — - — -) = i ' (3.1.5) 

K ' K 2 ; {k- 1 ^ Case 2. V ' 

3) (eigenspace ofir(wo)) vr(wo) acts on Hi ^ by the scalar (— l)"^*^ 2~~ . 

4) (intertwining operator) Fix any 4> G ^{'(IRP" 1 ), V £ 3f fe (IR 9_1 ), and 

ui tk (ru,rr]) := fi )k (r)(p(uj)ip(r]) G (3.1.6) 



6G 



Then 7 : L (C) — > V p ' q has the following form on the subspace H^ k : 

Tu ltk = c,^,,^)/^^^ (VO, (3-1.7) 

where 1^ : ^(M™" 1 ) -> ^'(IR m ) (0 < i < j) is an 0(m-l)-homomorphism 
defined in Fact \ 7. 5l\ and the constant is given by 



{ T^+l-k + l) 



-jl+k 



2 £_£ +; _ fc+3r(2 _2 + z) 



Case 1, 



-± — Case 2. 



(3.1.8) 



5) (L 2 -norm) For any (l,k) E N 2 



ll Tu ^H" 2 (M) = 2aa,it)+p-2 IKfe|l ^( c )- (3 - L9) 

A special case (the k = case) of Theorem 13.1.11 (2) provides us an 
explicit i'f -finite vector for every if-type. 



Corollary 3.1.2. For each a E N, the function 

9-2 

2 

is a K -finite vector for any </> G ^(M^ -1 ). More precisely, it belongs to the 



■K 3 -2(2r)(j){(jo) (3.1.10) 

Rnzie vector for any <j) G J- 



3i a (]R p ) ® 3i aH 2-(IR 9 ). 

Proof. Under our assumption p > g, o(Z,0) = I by the definition (|3.1.5|) 
because (1,0) belongs to Case 1. Hence, Corollary follows immediately from 
Theorem [3X1] (2). □ 

By using the unitary inner product ( , )m (see (|2.1.7|) ) for the conformal 
model (uj p ' g ,V p ' q ) of the minimal representation G = 0(p,q), the formula 
(|3.1.9p in Theorem 13.1.11 (5) can be restated as 

(7ui )k ,7ui jk )M = 2lN,fclli 2 (C)- (3.1.11) 

Remark 3.1.3. Theorem\MJ\ (2) for I = k = (or Corollary |XO| for 
a = 0) was proved in Theorem 5.8]. Since p > q, we are dealing with 
Case 1 if I = k = and a(0, 0) = 0. In this particular case, Theorem \3.1.1\ 
(2) asserts that /oo( r ) = r ^ r Kq^z(2r) belongs to the minimal K-type 

' 2 

< K°(W)® ( K E ^ L (W) of(Tr,L 2 (C)). 
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Remark 3.1.4. For q = 2, ir is essentially a highest weight module (see 
Section \1.7\ for the n-i = 1 case). We note that for q = 2, !K k (M q ~ 1 ) is non- 
zero only if k = or 1 (see Appendix 7.5). Thus our assumption p + q > 6 
combined with q = 2 and k = 0, 1 implies + I — k = — 2 + I — k > 0. 
Hence, the pairs (I, k) (k = 0, 1) belong automatically to Case 1. It then 
follows from the definition (3.1.2) that 

fl,o(r) = A +l K_i(2r), 
/i,i(r) = rKffi(2r). 

By using the formula 
("see (j732j) J, we aei 

//,oM = / i ,iW = ^r l e - 2 »-. 

VFe note that fi t o(r) (= /j i(r)) coincides with the function l fij(r)' in $()[ 
(3.2.4)] U P to a constant multiple. (In |^6] /, we investigated the Schrddinger 
model for 0(p, 2), and i/ie function 'fi i(r) ' was defined by using the Laguerre 
polynomial. The definition of 'fu(r) ' is different from (|3.1.2|) . ) 

Our method to show Theorem 13.1.11 is based on the technique of [49|, 

§5.6, §5.7]. The key lemma is Lemma 17.8.1 1 which gives a formula of the 

Hankel transform of the i^-Bessel function with trigonometric parameters 

by means of the Gegenbauer polynomial. 

The subspace Hi^ is not dense in L 2 (C), but is large enough (see 
l,ken 

Section that we can make use of Theorem 13.1.11 for the proof of Theorem 
14.1.11 in Chapter |U (see also Section 4.11 for its idea) . 

3.2 K fl M max - invariant subspaces H^ k 

The subspace ® Hi f. is not dense in L 2 (C), but serves as a 'skeleton'. In 
this section, we try to clarify its meaning. 
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We begin with the branching law G { K (see (|2.1.6|) ) and K { K' [K 1 
denotes K' := K n M™ ax ~ Q(p - 1) X 0(q - 1)): 



£ 2 (<?k ^ M a (MP) ® 0i a+ V (R«) (3.2.1) 

a=0 

oo a 2 

-00 M'fr-^^'fr" 1 ). (3.2.2) 

a=0 Z=0 fc=0 

The irreducible decomposition p.2.ip shows that L 2 (C)k is multiplicity- free 
as a A'-module. Hereafter, we identify the K-module "K a (W) ®"K a+E ~^~ (R q ) 
with the corresponding subspace of L 2 (C)k- Then we observe: 

51 (K'-type) Fix a pair (l,k) £ N 2 . In light of (|3Xl ) and (3.2.2), the fT'- 
type J{ l (W~ 1 )^J{ k (R C1 - 1 ) occurs in the K-module IK a (IR p )® 5f° +£ r a (M 9 ) 
if and only if a > a(£, fc). Further, Hi^ is characterized as a subspace of 
L 2 (C) satisfying the following two conditions: 

J W ~ K^RP' 1 ) ® ^{RO- 1 ) as K'-modules, 
1 W C Oi a( - l ' k XRP) <g) 5f°( i ' Ai ) +2 i a (R3). 

52 (K-type) Fix a £ N. Then for (Z,fc) £ N 2 , 

( n (^(K p ) ® ^ a+E ^(M 9 )] 

is non-zero if and only if at least one of I and /c attains its maximum in 
the set {(/', k') £ N 2 : < V < a, < k' < a + or equivalently, in 

the set of the K'-types (I', k') occurring in ^{W) tg> "K^^ (R q ) (see 
the black dots • in the figure below). 
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3.3 Integral formula for the K n M max -intertwiner 

As a preparation for Theorem 13 .1.11 we rewrite the integral formula for the 
G-isomorphism 7 : L 2 (C) -» Wv(c L 2 (M)) (see Fact ]2XT]) applied to 
spherical harmonics. The main result of this section is Lemma 3.3.1. 
We write v G M = S*^ 1 x S^ 1 C as 

v = \v , v', v",v p+q ^), v , v p+q ^ G K, «' G Rf" 1 , u" G M 9 " 1 (3.3.1) 

satisfying 

2 i I /|2 I // 1 2 i 2 i 

Lemma 3.3.1. Suppose that u G L 2 {C) is of the form 

u{ru,rq) = f(r)(j)(uj)ip(r]), 

for <p G ■X l (W>- 1 ), V G "K k (R q " 1 ), and f G L 2 (R + , r p+q ~ 5 dr) with regard to 
the polar coordinate (|2.2.2p . Then, 7u is reduced to the following integral 
transform of one variable: for v G M such that vq + v p + q -i > 0, 

V=T(i +fc)T ^ 2 -y" 2 ( V' \ 

(7u)(v) = e 2 LJ — ^ — ^ — U 

v + v p+q -i \\tr\J 

x f/M^,/^-)^,^^-)^*, (3.3.2) 
Jo 2 + \v + V p+q -i J 2 ^ K \v + v p+q -xJ 
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where J u (z) denotes the Bess el function. 

Proof. We recall from (pETTD that 7 = o o T. Let us compute 

(3"" 1 o Tu)(z) for z e By the formula l|2T5j) and (|2T2T6l) for Tu = 

u5(Q), we have 

2{J- l oTu){z) 

= (2 7 r)- 2± i z ^ /°° / /" /(r)(/»(cu)'0(r/)e^ v/3T(<2 '' rw)+<2 "' r ' ?)) r p+9 - 5 ( ir^d7 ? 



r^r-{l+k), n-s=l, //,-2^3 / z' \ ( z" 

/■oo 

x / /(r)J e= 8 + ,(r|2/|)J a= 3 +fc (r|z"|)rH-dr. (3.3.3) 
jo 2 2 

Here, in (|3.3.3p . we used the following formula (see e.g. [3H Introduction, 
Lemma 3.6]): 

71 , J i-i+^W 



e^^^dco = (2 7 t)t v ^IV(??) 11 ■ (3-3.4) 

Then, by the definition ()2.1.9p of the pull-back (^*) _1 , we get the desired 
result (15X21) . □ 

3.4 i^-finite vectors fi^ in L 2 (C) 

In this section, we collect basic results on the real analytic functions fi^ 
defined in (|3TP]) . 

Lemma 3.4.1. / z>fe G L 2 (M+, ir p+9 " 5 dr). 

Proof. The i^-Bessel function decays exponentially at infinity. The asymp- 
totic formula (see Fact 17.2.11 (2) in Appendix) implies 

, , \ \ c r _£ 2 1+ 'e~ 2r Case 1, . , , 

Mr)~[ cr .^ ke _ 2r Case2 ; (3-4.1) 

as r — > 00. On the other hand, since K u (r) = 0{r~ 2u ) as r tends to (see 

f0(r<-*-s+3) Casel, 

as r tends to 0. In either case, 

A* = 0(r^ +3 ) by the definition (^XT|) of 
Cases 1 and 2. Hence, we have fi :k e L 2 (M + , \rP +q ~ b dr) for p + g > 4. □ 
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We shall give a finer estimate on the derivatives of fi^ in Lemma 4.4.1, 
The explicit formula of the L 2 -norm of fi^ is obtained by the integration 
formula (7.2.13) of isT-Bessel functions as follows: 

Proposition 3.4.2. 



Case 1, 

(3.4.3) 

Case 2. 



ll/^*=llL2( K+i i r y+ a -5 dr ) 

r(^i + z) 2 r(2±f^ + 1 + /t)r(£^±2 + i- k ) 

i6r<> -i + 2/) 
r(^ + fc) 2 r(g±|^ + k + /)r(^£±g + fc _ z) 

16r(g - 1 + 2k) 
Here is another integral formula that we use later: 
Lemma 3.4.3. 

p-\-q — 6 , i — T - 

+v ~ fi,k( r ) dr 

_ f lr(2±f=^ + i+k+v=i* )T{ P^L + i-k+i+v=ix j Case lt 

" 1 3r(2+|=4 + W +^ )r( a + fe-f+l+V=Tx ) Case & 

Proof of Lemma \3.4-3[ Apply the formula (|7.2.1ip of the Mellin transform 
of -RT-Bessel functions. □ 

In order to compute Tu^ explicitly by using the integral formula (3.3.2), 
we need yet another lemma: 

Lemma 3.4.4. For a pair (l,k) G N 2 ; let fi ^ be the function on C defined 
in (|3.1.2p . With respect to the coordinate t (vQ, v' , v", v p + q ~i) G M C W +q 
(see [3.3A§), the integral 

I" flArV^J-^—) J^J^—y^dr (3.4.4) 
is equal to: 

2 (v + v p+g ^)\v'\ 2 +l \v"\ 2 + (v p+q ^) Case 1, 

r(- E + 2 - / + + 1) , „P-3,,, ,,,8-3 , , -STl+l 
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Proof of Lemma \3.4-4\ We treat Case 1 first. By the change of variables 
t := 2r, the integral (|3.4.4p amounts to 

1 f°° p-i,, T / |r/|t \ / |w"|t \ r ^ 

Applying Lemma 17.8.11 with 

M : = - 2 ^ v := — 2 cos0:=u o , cos0 := ?;p +9 _i, 

we get the formula in Case 1. 

The proof for Case 2 goes similarly. In this case, the integral amounts 

to 

1 f°° a=l +k T / W\t \ ( \v"\t \ 



by the change of variables t := 2r. Now, we substitute \x := + ^> 
+ /, cos# := and cos</> := vq into (|7.8.ip . □ 



: = 



p-3 
2 



3.5 Proof of Theorem I3TTTT 



In this section, we complete the proof of Theorem 13.1.11 
Proof of Theorem \3.1.1[ 1) By the isomorphism 

L 2 (C) ~ L 2 (M+, Ir p+ "- 5 dr) g L 2 (S P " 2 ) g L 2 (5 9 " 2 ) (see 
in the polar coordinate, the first statement follows immediately from Lemma 

ED 

4) By ([3X2"]) and Lemma E331 we have 



. %k T{^ + l)\A l ^)\A k ^^)C^ k (v P+q -i) Case 1, 
ci, k Y{^ + fc)| u '|^(^)C_^ +i _ fc) ( W o)b / ?V(^) Case 2, 



where c^j- is the constant defined in (I3.1.8P and v = t (vo, v', v" , v v+q -\) € 
S p-i x 5-9-1 Now we uge the definition (|7Xll that 
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for (xo,x') G S m 1 , and in particular, for i = j 

' m — 2 \ . ,,„■,/ x' 



by (17X51) . Thus, the formula (|3XT1) follows. 

2) The first statement is obvious. Since I^L,j maps ^(M" 1 " 1 ) into 
W(R m ) (see Fact EXT}, (|3X7j) implies 

Hence, we have proved the second statement. 



3) By (|2.1.5|) . the unitary inversion operator vj p ' q {wo) on V p ' q is given 

by 

w p ' q {wo)h ) (v , v") = h(v',—v"). 



On the other hand, it is easy to see h(—x) = (— l) J /i(x) for h G ^(M 9 ) (see 
Appendix 17.51 HT|). Therefore, TZ7 p ' 9 (u;o) acts on each if-type component 
3{ a (W) ® IK a+E ^(M 9 ) by a scalar multiple (-l)*^ 2 ?. Since 7 : L 2 (C) 
V p ' q intertwines ir and 7r(u>o) acts on ffyj. as the scalar (— l) a+E 2 1 

because H lyh belongs to the K-type K a ( l > k \W) ® ft^+^R 9 ). Thus, 
the third statement is proved. 

5) We claim that the following formula holds: 

IITu^H^^p-lx^-l) = h )k T{k + I + - 2)||0||£2 (SP -1) 11^1112(5,-1), 

(3.5.1) 

where 

'7rr(p-2 + 2/)r(^ + /-A; + l) „ i 
— tttt; Case 1, 



2 2p+4/ r ( / + £)2 

^r(g - 2 + 2k)T(k - I - ^ + ^ n 

— Case 2. 



To see this, we begin with (j3.1.7p : 

H^,fclli2(SP-ixS9-i) = l c ^fc| 2 |l I La(/,fc)(^)lli 2 (5P- 1 )ll I fc^ a ( i>fc ) + £^(^)lli 2 (5<'-i)- 
By (3.1.7) and the norm formula (7.5.2), the right-hand side amounts to 



l2 2 3 - p - 2l 7rT(p-2 + l + a(l,k)) u ^ 2 
M (a(l,k)-l)\(a(l,k) + ^ mL2{Sp - 



2 3 - q - 2k irT(q - 2 + k + a(l, k) + ^ , 



(a(Z, k) + ^- k)!(a(l, k) + 2=2 + 



2tH'^IIl 2 (S'J-1)- 



74 



Now, substituting (|3.1.5|) and (|3.1.8p into this formula, we get (|3.5.ip by 
elementary computations. 

Finally, comparing (3.5.1) with Proposition 3.4.21 we obtain (|3.1.9|) by 
the Gauss duplication formula (|7.4.3p of gamma functions. Hence, Theorem 
13. 1 . 1 1 is proved. 

□ 
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Chapter 4 

Radial part of the inversion 



The goal of this chapter is to find the 'radial part' TJ ^ of the unitary inversion 
operator $c '■ L 2 (C) — ► L 2 (C). The main result of this chapter is Theorem 

am 

4.1 Result of this chapter 

Suppose p > q > 2 and p + q > 6. We recall from Section E2] that the polar 
coordinate of the (generalized) light cone C: 

R + x S p ~ 2 x S q ~ 2 ~ C, (r, u, rj) ^ (ru, rrj) 

induces the isomorphism of Hilbert spaces: 

L 2 (C) ~ L 2 {R+, K p+q - 5 dr) § L 2 {S p - 2 ) g L 2 (S q ~ 2 ). 

This isomorphism respects the action of the compact group 

K' :=KD M^ ax ~ 0(p - 1) x 0(q - 1). 

Since the Hilbert space L 2 (S P ~ 2 ) is decomposed into the direct Hilbert sum 
of spherical harmonics (see H6 in Appendix 7.5): 

oo 

L 2 (S P ~ 2 ) ~ ^0* p_1 ) j 

j=0 

and likewise for L 2 (S q ~ 2 ), we have a decomposition of the Hilbert space 
L 2 (C) into the discrete direct sum: 

oo 

L 2 (C)~ ^®L 2 (IR + ,-r p+<? - 5 dr)®^(M^ 1 )®^{ fe (]R 9 - 1 ). (4.1.1) 

Z,fc=0 
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Each summand of (|4.1.ip is a if'-isotypic component. 

For each (l,k) £ N 2 , we introduce real analytic function Ki k(t) by 



\ A( 1 V+ E i a r , 20^2|i+fc -g+3+l-k -p-q+6-l-k -p+3-l+fc \ p -, 

lM> ' \ 4 (-l) fc G 2 °(t 2 |^, -v-if- 1 -* , ^+\+ l ~ k ) Case 2. 

(4.1.2) 

Here, G^(x\bi, 62, 63, 64) denotes Meijer's G-function (see Appendix 17.61 for 
definition). For the definition of Cases 1 and 2 with regard to the parameter 
(l,k) £ N 2 , we recall (j3.1.1|) . namely, 

Case 1: 2=2 + I - k > 0, 

Case 2: 2=2 + / _ k < . 

The above formulas give the same result in Cases 1 and 2 if 2=2 _|_ / _ 
k = 0. Later, we shall give an integral expression of Ki ^ by means of the 
Mellin-Barnes type integral (see Lemma I4.5.2p . The latter expression is 
more natural in the sense that the formula is independent of Cases 1 and 2. 

Theorem 4.1.1. 1) The unitary inversion operator $c = k{wo) preserves 
each summand of (|4.1.1j) . on which tt(wq) acts as a form I/ fc® id (8) id. Here, 
T^k is a unitary operator on the Hilbert space L 2 (M + , ^r p+q ~ 5 dr). 

2) For each l,k G N, the unitary operator 1} & is given by the integral 
transform against the real analytic function K^f. (see (|4.1.2p for definition): 

(Ti,kf)(r) = - / K ltk (rr')f(r'yP + i- 5 dr'. (4.1.3) 
1 Jo 

Remark 4.1.2 (Case q = 2). Theorem \4-l-l\ covers the case q = 2, where 
the representation tt is essentially a highest weight module (see Section \1.7\ 
for the ri2 = 1 case). In this case, J{ k (M}) is non-zero only if k = or 1 
(see Appendix \ 7. 5\ for convention). As we saw in Remark \3.1.4[ the pair 
(/, k) belongs to Case 1 for any I G N because the inequality 2=2 _|_ / _ & > 
is implied by p > 6 — q = 4. Hence, by the definition (|4.1.2p of Ki t k(t), we 
have 

K lfi (t)=4(-1)^G^ 1 1 + 1 ~ P + A ~ l -^ + 3 ^ 



2' 2 ' 2 2 



i+e^2 20/ + 2i* + 1 1 -P + 3-Z -p + 4 



2 ' 2' 2 

In view of the symmetric property of the G-function: 

G04OI&I, &2, &3, 64) = Os|&2, & 4, 6, 
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the above formulas show Kip(t) = K^i(t). Applying the reduction formula 
(|7.6.14p of the G-function of the form Gol(x\a, a + i, b, b + \), we get 

Ki, (t) = K lA (t) = ^-ly+^rf J p _ 3+2l (4Vt). (4.1.4) 

Thus, the integral transform I] & (k = 0, 1) is the Hankel transform on R + 
(after a suitable change of variables). Therefore, Theorem \4-l.l\ in the case 
q = 2 gives the same result with f4~6\ Theorem 6.1.1], but the proof here is 
different from that of \4°1 . 

Remark 4.1.3 (Comparison with the Weil representation). We record here 
the corresponding result for the Schrodinger model L 2 (M n ) of the Weil rep- 
resentation w of Mp(n,M) (see also \4 6\ Remark 6.1.3]). We adopt the 
same normalization of the Weil representation with \38^ . Then, the unitary 

inversion operator w(w ) = e 4 3"m«- 

1) According to the 0(n)-isotypic decomposition of L 2 (IR n ) 

oo 

L 2 (M n )~ ^®L 2 (M + ,r n - 1 dr)®^(M n ), 
z=o 

the unitary inversion operator zu(w' ) decomposes as 

oo 
1=0 

by a countable family of unitary operators T[ (I S N) on L 2 (R + ,r n ~ 1 dr). 

2) The unitary operator T[ is given by the Hankel transform 

/•oo 

(T//)(r) = / K' l (rr')f(r f )T fn -W, 
Jo 

where the kernel K',(t) is defined by 

K'At) := e s n t a~ J „- 2+2i (t). 

2 

Returning to Theorem I4.1.1| we remark that the group law Wq = 1 
in G implies 7r(u;o) 2 = id, and consequently T 2 fc = id for every I, k G N. 
Further, as ir(wo) is a unitary operator on L 2 (C), so is its restriction on 
L 2 (M_|_, ^r p+q ~ 5 dr) for every I, k. Hence, Theorem 14.1.11 gives a simple group 
theoretic proof for the Plancherel and reciprocal formulas on the integral 
transform involving the G-functions due to C. Fox [19j : 
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Corollary 4.1.4 (Plancherel formula). Let 61, b 2 , 7 half-integers such that 



b\ > 0, 7 > 1, < 62 < 5 + b±. Then the integral transform 



1 



00 



^61,62,7 : f( x ) ~ / G 0A ({xy)'y\b 1 ,b2,l - 7 - hA - 1 - b2)f(y)dy 
7 Jo 

is a unitary operator on L 2 (M + ). In particular, we have 

\\ S b 1 ,b 2 , 1 f\\L2(R+) = II/IIl 2 (m + )- (4.1.5) 

Proof. Set 61 := *±*, b 2 := " 9+3 2 + '" fc , 7 := 2±§=*, x = r 2 ^, y = r /2 ^. 
Then, the assertion is equivalent to the fact that 7} & is a unitary operator 
on L 2 (B. + ,±rP +q - 5 dr). □ 

Corollary 4.1.5 (Reciprocal formula). Retain the notation and the assump- 
tion as in Corollary \4-l-4l Then, the unitary operator S , 6 1 ,b 2 ,7 ^ s of order two 
in L 2 (IR_|_), that is, the following reciprocal formula holds for f £ L 2 (R_|_): 

(S blM ^f)(x) = - / G 2 4 ((xy) 7 |6 1 ,6 2) 1 - 7 -6 1} l -J-b 2 )f(y)dy 
7 j 



1 



OG 



f(x) = - I Gf A {(xy) 1 |6i,62,l -J-h, 1 -7- b 2 ){S hu b 2 ,r 1 f){y)dy 
7 Jo 

Remark 4.1.6 (Fox's reciprocal formula). The reciprocal formula for the 
G-transform was found by C. Fox fWjj for the following generality: 

a\, ■ • • , On, 1 - 7 - a\, ■ • • , 1 - 7 - a n 
61, • • • , 6 m , 1 - 7 - 61, • • • , 1 - 7 - b m 



O ' r 

LT 2n,2m \ x 



Corollary \4-l-5\ is a special case of Fox's formula corresponding to the case 
(m, n) = (2,0). On the other hand, it follows from Remark \4.1.S\ that Corol- 
lary \4-l-~5 in the special case when q = 2 yields the classic reciprocal formula 
of the Hankel transform (T7P| §14-3], see also fjo)/ for the connection with the 
representation of the conformal group 0(p,2)). Our approach gives a new 
representation theoretic interpretation (and also a proof ) of these formulas. 

Remark 4.1.7 (Fourth order differential equation). Let ft be the Casimir 
element for the Lie algebra t. Since an element of the Lie algebra n max acts 
on smooth vectors of L 2 (C) as a differential operator of second order (see 
(|2.3.16p ). the Casimir operator dir(Q,) acts as a differential operator on C 
of fourth order. Let us examine what information on Kik(t) we can obtain 
from the Casimir operator dn(fl). 
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Since Ad(wo)£l = £1, we have the commutative relation 

7t(wq) o dir(Q) = dir(Q) o ir(wo). (4.1.6) 

On the other hand, fi commutes with all the elements in I, in particular with 
6'. This implies that the differential operator dir(^}) preserves each summand 
of (|4.1.ip . and the identity (|4.1.6p can be regarded as the identity on each 
summand of ()4.1.ip . Hence, ()4.1.6p leads us to the fact that the kernel 
function Ki^(t) for tt(wo) solves a certain differential equation of order four 
for each (l,k). In view of the formula (|4.1.2H of K\^, this corresponds to the 
fact that Meijer's G-function Gq^(x | 6i,fr2>&3>&4) satisfies the fourth order 
differential equation (see (|7.6.6p ): 

4 , 

n(^- & >=°- 

3=1 

For q = 2, the situation becomes simpler because the minimal represen- 
tation 7r is essentially a highest weight module. In this case, the Lie algebra 
t = So(p) ®so(g) contains the center so (g) = so (2). Taking a normalized 
generator Z in so (2), we can show 

d7r(0) = dn(Z) 2 + constant . 

We note that the differential operator dir(Z) is of order two, and this in turn 
corresponds to the fact that the kernel function reduces to the Bessel 
function (see (|4.1.4p ) solving the Bessel differential equation of order two 
(AppendixYnty- 

The rest of this chapter is devoted to the proof of Theorem 14.1.11 The 
key properties of the element wq and the unitary inversion operator ir(wo) 
that we use in the proof are listed as follows: 

1) wo commutes with K' . 

2) Ad(w )H = -H (see ([232])). 

3) 7r(w )\ Hl>k =±id (Theorem EXU (3)). 

The first property (1) gives immediately the proof of Theorem 14.1.11 (1) 
(see Section l4~2j) . 

The second property (2) implies that the kernel function of the 
unitary operator T\^ is a function essentially of one variable rr' (see Section 

KM- 

The non-trivial part of Theorem 14.1.11 is to prove that this one variable 
function is given by Meijer's G-function Gq®, namely, by the formula ( 4.1.2j) . 
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The property (3) is used for the proof of this part. Our trick here is based 
on Fact 14.3.1] which assures that the desired formula (|4. 1 .3|) holds for any 
function / once we can prove (|4.1.3p holds for a specific function / = fn~ 
(a (K, K')- skeleton function). Actual computations for this are carried out 
in Section 14.61 A technical ingredient of the proof is to show that is a 
tempered distribution (see Proposition 4.5.6). 

Summing up the formulas for over I, k, we shall give a proof of 
an integral formula of tt(wo) for an arbitrary L 2 -function on C in the next 
Chapter [5] (see Theorem 15. 1 . 1 [) . 

4.2 Proof of Theorem \4AA\ (1) 

As we have explained at the beginning of Section I4.1|, the decomposition 
(|4.1.ip of the Hilbert space L 2 (C) corresponds to the branching laws of 
the restriction of the unitary representation (tt,L 2 (C)) from G to K' ~ 
0{p — 1) x 0(q — 1). Thanks to the group law worn = mwo (see 12 in 
Section [23]) . the relation 7r(u>o) o n(m) = n(m) o ir(wo) holds for all m E K'. 
Therefore, the unitary inversion operator ir(wo) preserves each summand of 
the decomposition (|4.1.ip . 

We now observe that the group K' acts irreducibly on Jf'(IR p_1 ) (g> 
< K k ( 1 R q ~ 1 ) (see Appendix 17. 51 H2). Therefore, it follows from Schur's lemma 
that 7r(wo) acts on each summand: 

L 2 (M+, K p+q - 5 dr) (8) ^(W- 1 ) ® M fc (M 9_1 ) 

as the form Tj^® id (8) id for some operator on L 2 (M + , ^r p+q ~ 5 dr). Such 
an operator Tn- must be unitary because n(wo) is unitary. Now Theorem 
14.1.11 (1) has been proved. □ 

4.3 Preliminary results on multiplier operators 

We recall the classic theory of multipliers on R, followed by the observation 
that the multiplier is determined by the action on an (appropriate) single 
function. 

We write l(t) (t E M) for the translation operator on L 2 (M), namely, 

{l{t)F)(x) := F(x-t) for F G L 2 (R). (4.3.1) 
An operator B on L 2 (M) is called translation invariant if 
B o l(t) = l(t) o B for any t G R. 
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We write S(M) for the space of rapidly decreasing C°°-functions on M 
(the Schwartz space endowed with the Frechet topology), and §'(M) for its 
dual space consisting of tempered distributions. Then, the Fourier transform 
J : S(R) -> S(M), ^ ^ i J R5 (x)e v/3T:r «dx (see (TL2TT]) ) extends to S'(M) 

by 

(U,g) = (U,3g) for C7 G S'(M), 5 G S(R). (4.3.2) 
For 17 E S'(K) and / G 8(1), we have 

l(t)(U * f) = U *{l{t)f) foranytGlR. 

Hence, / 1— > [/*/ is translation invariant. Furthermore, if GFi7 is a bounded 
measurable function, then / 1— > {7 * / extends to a bounded operator on 
L 2 (M) and its operator norm is given by V27r||977||£,°°(]R) because 



\\u * f\\mm = \\m * f)\\L*(R) = V2n\\(?u)(?f)\\ L 2 m . 

Conversely, the following theorem also holds: 

Fact 4.3.1. Let B be a bounded, translation invariant operator on L 2 (M). 
Then we have 

1) There exists a unique tempered distribution U whose Fourier trans- 
form STJ is a bounded measurable function such that Bf = U * f for any 
f£L 2 (R). 

2) //, moreover, Bfo = f\ for some function /q G L 2 (IR) such that 
S/oCf) + for a.e. £ G M, then U = -^^(f*). ' 

Proof. 1) See Stein- Weiss \68\ Chapter I, Theorem 3.18], for example. 
2) It follows from Bfo = f% that 3(17* /o) = 9\fi, and therefore we have 



Hence, the bounded measurable function STJ is determined by the for- 
mula 

mo- 1 mo 



2ir?fo(0' 

□ 



Next, we introduce two linear maps cr+ and cr_ by 



a + : L 2 (R + , ±r*>+*- 5 dr) - L 2 (M), /(r) ^ -^e^*/^), 

p+<?-4 ^ „, (4.3.3) 



CT— : L 2 (M+, irP +< ?~ 5 (ir) -» L 2 (M), /(r) ^ 4= e - Iir ^ ii 7(e~ !B )- 
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Then, both a + and ct_ are unitary operators. Further, clearly we have 

(a.f)(x) = (a + f)(-x). (4.3.4) 
The inverse map aZ is given by 



(aZ 1 F)(r) = v / 2r~ 2L -i— F(-logr) for F G L 

We define a subspace S of L 2 (R + , \r p+q ~ b dr) by 

S := ctZ 1 (8(E)) = a+ 1 (§(R)), (4.3.5) 

and endow S with the topology induced from that of the Schwartz space 
S(R). Now let §' be the dual space of 8. 

Since <r_ is unitary, we get the following identity for F G L 2 (IR) and 
it G L 2 (M + ,irP+5- 5 (ir): 



(o-_u,-F} L 2 {]R) = / (cr_-u)(x)F(x)(i2; 

JR 

= (u,a_ -F 1 ) L 2( R+i i r . P+9 -5 dr ). 

Then a_ extends naturally to an isomorphism from the dual space §' 
onto S'(IR) by the formula 

(a-u, F) := (u, aZ 1 F) for F G S(R), u G §'. (4.3.6) 

Recall from (13.1.2P that we have defined a family of real analytic func- 
tions fik on R + parametrized by (l,k) G N 2 . As we saw in Lemma 13.4.14 
f l)k G L 2 (R + , \rP + ^dr). 

For a continuous operator A on L 2 (R+, ^r p+l?_5 (ir), we define a contin- 
uous operator j4 on L 2 (R) by 

i^C^o^o^ 1 . (4.3.7) 

Thus, the following diagram commutes: 

L 2 QR+, ^r p+q ~ 5 dr) L 2 {R + ,K p+q ^dr) 

a+i <r_ { (4.3.8) 

L 2 (R) L 2 (R). 

Since a± are unitary operators, A is unitary if and only if A is unitary. 
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For k £ §', we define an operator A K by 

A K :§^ §', /(r) ' ► - / ^CrrO/CrOr'^ 5 *'. (4.3.9) 

It follows from the definition (4.3.3) of 0"+ and cr_ that 

1 -t- — 4 r 00 

(a_K * <t+/)(:e) = -e-^ x / e^+^Me"^)/ (e^dy. 

^ J -oo 

Then, by the change of variables, we have 

AJ = -^=aI 1 (a^K*a + f) for / G §. (4.3.10) 

The following lemma characterizes operators of the form A K : 

Lemma 4.3.2. Let {p(t) : t £ IR} 6e a one parameter family of unitary 
operators on L 2 (M + , ir p+c/ ~ 5 <ir) defined by 

(pWM-e-^tfie-'r). (4.3.11) 

Suppose that a unitary operator T on L 2 (M + , ^r p+q ~ 5 dr) satisfies the fol- 
lowing (anti-) commutative relation: 

Top(t) =p(-t)oT foranyt£R. (4.3.12) 

Then, there exists a unique distribution k £ §' such that T = A K . 

Proof. For a general operator A on L 2 (M + , ^r p+l?-5 <ir), we shall use the 
symbol A to denote o Ao a^ 1 as in (|4.3.7p . Then we have 

T o p(t) = <t_ o (T o p(i)) o (j^ 1 = T o (a + o p(t) o cr^ 1 ), 
p(—t) o T = o~_ o (p(—t) o T) o er^ 1 = (o~_ o p(—t) o aZ 1 ) o T. 

On the other hand, by a simple computation, we have the following identi- 
ties: 

a + o p(t) o o"T = cr_ o p(— t) o (tZ 1 = /(£). 

Here, l(t) denotes the translation operator (|4.3.ip . Hence, the relation 
(|4.3.12p is equivalent to 

fol(t) = l(t) of for any t £ 1, 
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that is, T is a translation invariant bounded operator on L 2 (M). Therefore, 
the operator T must be a convolution operator U * for some tempered distri- 
bution U £ §'(R) such that its Fourier transform 9T7 is a bounded function 
(see Fact ELD (1)). 

Finally, by setting k := y/2aZ l U, we have for any / G L 2 (M + , ir p+9_5 <ir), 

Tf = aZ 1 ofo a + f 

= ^=<rZ 1 (<T-K*a+f) 

= 4./ 

by (|4.3.1U|) . Therefore, T = □ 



4.4 Reduction to Fourier analysis 

In this section, we apply the results of the previous section to the uni- 
tary operator 2]^ on L 2 (M + , ^r p+q ~ 5 dr). The main result of this section is 
Proposition 14.4.41 

We begin with a refinement of Lemma I3.4.1L 

Lemma 4.4.1. f^f. belongs to 8. In particular, f^f. G L 2 (M_|_, ^r p+l?_5 (ir). 

Proof. By the definition of S (see (|4.3.5|) ). it is sufficient to show cr-fi^ G 
S(M). The proof is divided into two steps. 

Step 1: For any m£N, x m (a-fi ) k)(x) is rapidly decreasing. 

By the definition (|3.1.2j) of fa-, we have 

1 , d+o „n I Kq-z . ,(2e~ x ) Case 1, 

= ^=e-(^^- 2 ^ x ^ _ J ' (4.4.1) 

V2 Kp-3 ,(2e x ) Case 2. 



2 



Therefore, by the asymptotic behavior of A'-Bessel functions (see Fact 7.2.1 
1), 2), respectively), we get 

, , w , e-* fr(2^ + A:)e-( £ ? + '- fc ) a; Case 1, 
2V2 + l)e { ~ +i L)x Case 2, 

(4.4.2) 

, , w , 1 fe-f^+O^e-e- Case 1, 

(a -^* )(a!) ~ 7^ X \ e -(^+*)» e -- Case 2, &S * ^ 

(4.4.3) 
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Thus, Step 1 is proved. 

Step 2: For any neH, ^ 7r (cr_// j / c (x)) is rapidly decreasing. 

Let us prove the above statement using induction on n. We already know 
from Step 1 that the above statement is true for any l,k G N in the case 
n = 0. Now assume that the statement is true for n. Then the statement 
for n + 1 immediately follows from the next claim. Now Step 2 has been 
proved, and so has Lemma 14.4.11 □ 

Claim 4.4.2. For any I, k G N, we have the following recurrence formula: 

^-fl,k) = -( P + l~ 4 + I + k)a-fi, k + 2<7_/, +1>fc+ i. (4.4.4) 
Proof. For simplicity, we rewrite the formula (|4.4.ip as 
{a-fi,k)(x) = -±=e~ ax K u (2e- x ), 



where 

P + q -4 j^ + k Casel, 

h t + k, v = < % 

2 I ^ + / Case 2. 



2 

By the recurrence formula (17.2. 1H of the ET-Bessel function, we have 

±{a_f hk ){x) = ^{-ae- ax K v {2e- x ) + 2e-( a+2 >K u+1 {2e- x )) 

= -a(o-fi >k )(x) + 2(o-_/ J+ i j jfc + i)(a;). 

Here, we have used the fact that (k, I) belongs to Case 1 (i.e. + / — k > 0, 
see (|3.1.ip ) if and only if (k + 1, 1 + 1) belongs to Case 1, and likewise for 
Case 2. Therefore, we have proved Claim 14.4.21 □ 

Remark 4.4.3. The above proof for fa- G 8 was straightforward from the 
asymptotic behavior of fi )k and its derivatives. We shall prove in Proposition 
4-5.6 that K\ k G namely, a-Ki k G S'(R). Unlike the above proof for 
flk G 8, this is not obvious from the asymptotic behavior of Ki k (see Remark 
\4-5.1\ ). In fact, it follows from (|4.5.1|) that 

limsupe _€ ' :,: '((T_^ i fc)(x) = limsupe~ e ' x '~? (— %= cos(4e~t — ^ - n] 

x— >— oo x— >— oo ^v27T 4 

= +oo 

if e < |. Thus, the asymptotic behavior of a-K\^ is worse than that of 
any polynomial as x tends to —oo. As a result, our proof for K\ k G 8' is 
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more delicate, and is based on an explicit computation of the Fourier integral 
(see Proposition \4-5.6 ). We note that {a-Ki^){x) decays exponentially as 
x tends to +00. 

Proposition 4.4.4. Let 7} £ be the unitary operator defined in Theorem 
\4-l-l\ (I). Then, there exists uniquely a distribution Ki jk G §' such that 
Ti k = A Kl (see (|4.3.9|) for notation). Namely, we have: 



1 r°° 

(Ti,kf)(r) = - / K l)k (rr')f(ry^- 5 dr'. (4.4.5) 
1 Jo 

Proof of Proposition \4-4-4\ We recall from (|2.3.12p that the unitary opera- 
tor 7r(e tH ) on L 2 (C) can be written by means of the unitary operator p(t) 
(see (|4.3.11in on L 2 (M+, \r p+q ^dr) as follows: 



*(e tH ){f(r)<i>(uMr,)) = (p(t)/) (r)0(u,)^), 

where / E L 2 (R + , \r p+q ~ b dr), <f> £ ^(R* -1 ), and ij) £ 5C*(R« _1 ). 

Applying tt(vjq) to the both sides, together with the definition of 2} k 
(see Theorem 14.1.11 (1)). we obtain 

n(w )on(e tH )(f(r)^(w)^(r i )) = (T^ k p(t)f)(r)cP(w^( V ). 

Similarly applying tt(wo) followed by 7r(e~ tH ), we get 

7r(e-^)o7rK)(/(r)^H^(r?)) = (p(-t)T l>k f)(r)(l>{w)iP( V ). 

On the other hand, it follows from Ad(wo)H = —H (see f)2.5.2[> ) that 

woe tH = e~ tH wo, 

and then we have 

tt(w ) o Ti(e tH ) = ir(e~ tH ) o tt(w ). 

Therefore, 

Ti,k ° p(t) = p(-t) o T itk . 
Now, Proposition 4.4.4 follows from Lemma 14.3.21 □ 
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4.5 Kernel function K^k 



We defined a family of real analytic functions Kik(t) in (|4.1.2p by means of 
Meijer's G-function Gq®. This section studies basic properties of Ki^(t) as 
a preparation for the proof of Theorem 14.1.11 (2). The main result here is 
Proposition 4.5.6, 

We begin with an asymptotic estimate of Ki^it). 

Lemma 4.5.1 (Asymptotic behavior). Ka~{t) has the following asymptotics 
as t tends to or oo: 

1) As t tends to 0, 

, , \ 0{t~i +z+l ~ k ) Case 1 
K ' M ~ \o\t-^-») Case, ( " >2) ' 

Kl,t(t) = O(t') {q = 2). 

2) There are some constants P±, ■ ■ ■ , Qi, • • • such that 

K ltk (t) = 4= *~ ^ cos (^ " (1 + ^ + ^ + • • • ) 

V27T 4 



2p+2<j-9 . 1 2(7 — 3 x 1 3 

+ t 4 S in(4t 2 - ?r) (Qii~ 5 + ^2*" 5 +•••), (4.5.1) 

as t tends to +oo. 

Proof. Directly obtained from the asymptotic formula of Meijer's G-function 
Gq®{x j b\, &2 5 1 — 7 — &i, 1 — 7 — ^2) given in Lemma 7.6.4 in Appendix. □ 

Next, we give an integral expression of Ki^it) (t > 0), where the integral 
path L will be taken independently of I, k G N. We note that the integrands 
in (|4.5.2h and (I4.5.3P are the same. The first expression (4.5.2) is convenient 
in Case 1, and the second expression (4.5.3) is convenient in Case 2 (see 
Remark |4X3]). 

Lemma 4.5.2. Fix a real number 7 > — p+ ^~ 5 , and let L be a contour that 
starts at 7 — v 7 — loo and ends af 7 + V — loo aric ^ passes the real axis at a 
point so satisfying sq < — P+ Q~ e (see Figure \4.5. i| ). (Later, we shall assume 
also that — p+ ^~ 4 < so «n Chapter 5.) Then, we have 

(—-W^ 2 -^ r Y( l+k-s \-p( -g+3+l-k-s \ 

Klk (t) = [ / } fMl 2 J tMs (452) 



^-P= / + zzyj^ 2 , ,J+ fds. (4.5.3) 
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s-planc 




Proof. The equality (14.5.2P = (j4\5.3) is an immediate consequence of the 
following formula: 



j-i^ — q+3+l—k—s ' 



> p— l+l— k+s 

which is derived from 



p / q— 1 — l+k+s 
-p+3— l+k— s n 



("I) 



(4.5.4) 



r(*)r(i-*) 



7T 



SU17TZ 



Let us show (|4"X2"1) in Case 1, and (|433|) in Case 2 (see ([37TTT]) for the 
definition of Cases 1 and 2). As meromorphic functions of the variable s, 
the poles of the numerators in the integrands (|4.5.2p and (|4.5.3p are given 
by 

Wi := {I + k + 2a, -q + 3 + I - k + 2a : a £ N}, 
W 2 := {I + k + 2a, -p + 3 - I + k + 2a : a € N}, 

respectively. Then, 

infWi > _ p + q ~ 6 in Case 1 (i.e. ^^- + l-k>0), 



infW 2 >-^| — - in Case 2 (i.e. ^—^ + l-k<0). 
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Therefore, in either case, the contour L leaves all these sets W\ and Wi on 
the right because our L passes the real axis at some point sq < — ■ 

By the definition of Meijer's G-function (see (|7.6.2p in Appendix, see 
also Example I7.6.3P , we get (14.5.2P in Case 1 and fj4.5.3j) in Case 2 by the 
change of variables s := 2 A. Hence, Lemma is proved. □ 

Remark 4.5.3. We shall use the expression (j4.5.2H in Case 1 and (|4.5.3p 
in Case 2 later. The point here is that there is no cancellation of the poles 
of the numerator and the denominator of the integrand. For example, the 
poles of the denominator of the integrand (4.5.2) are given by 

Vy := {-p - q + 4 - / - k - 26, -p + 1 - I + k - 2b : b £ N}. 

Then, 

supT^i < inf W\ in Case 1, 

and therefore V\ n W\ = 0. Similarly, there is no cancellation of the poles 
of the numerator and the poles of the denominator of the integrand (|4.5.3p 
in Case 2. 

As Kik is a real analytic function on M + , so is (a^Kij ( .)(x) on M. (see 
(4.3.3) for the definition of 0"_), which in turn is a distribution on M. More 
strongly, we shall see in Proposition 14.5.61 that {a-Ki^){x) is a tempered 
distribution. 

For this, we define a meromorphic function ip{Q on C by 

r / p+g-4 , l+k-^TC \r(P-1 -i- l-k+l-%/^TC \ 

m : = (_iy+^iA ^ + 2 JM-r+ 2 ) (45 5) 

K J v ' r ( P+g~ 4 | /+fc+v / ^TC ^r;(P-g | Z-fc+i+y^TC ) 

= (-l) k 1 4 + 2 j 1 4 + (4.5.6) 

We shall use (I4.5.5P in Case 1, and (|4.5.6p in Case 2. The proof of the 
equality ( j4.5.5|) = (|4.5.6p is the same as the proof of the equality (|4.5.2p = 

Lemma 4.5.4. 1) |"0(C)l = 1 f or C G ^. /n particular, the inverse Fourier 
transform is defined to be a tempered distribution. 

2) ip(C) is a meromorphic function on C, and the set of its poles is 
contained in 

{— \J — lm : m = 1, 2, 3, . . .}. 
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3) For 7/1 < T] < 7]2, 



■17?)! 



-2'/ 



as |£| — > oo. 



(4.5.7) 



Proof. 1) Since T(z) = T(z) for z G C, we have IVKOI = 1- Therefore 
ip G S'(M) and thus r 1 ^ S'(K). 

2) The proof is straightforward from the definitions (j4.5.5j) and (I4.5.6P 
in each case. 

3) We recall Stirling's asymptotic expansion of the gamma function (see 
[TJ Corollary 1.4.4] for example): 



|r(o + V-Lb)\ = v / 2^|6| a -5 e -^(l + 0(\b\^)) 



(4.5.8) 



when a\ < a < ai and |6| — > oo. Then, for a G M and z = x + y/—ly 

(yi < y < 2/2), 



r(a 



-iz) 



r(a + 



-1*1 



\2y. 



l + 0(\x\ )) as |x| — ► 00, 



(4.5.9) 



where the constant implied by the Bachmann-Landau symbol O depends 
only on a, y\ and yi- Now, applying (|4.5.9p to z = — \/ — 1") twice, we 
get dOTjl . □ 

By the change of variable s = \/— TC ~~ , the integral formula of 

-Kz^ (Lemma I4.5.2P can be restated as follows: 

Lemma 4.5.5. Let 7 > — p+ ^~ 5 and L' 6e an integral path starting from 
— (7 + p+ 2~ 4 )v / ~l — 00 an d ending at — (7 + p+ 2~ 4 )\/— T + 00 passing the 



imaginary axis at some point in the open interval (- 
\4.5.%fy . Then, we have 



2 ' 



1 



7T 



, . . . P+q-4 

not —- 



or equivalently (see (|4.3.3|) for the definition of a-) 

1 



{(T-Klk)(x) 



V21T 



^(C)e- 



-1) ('see Figure 
(4.5.10) 

(4.5.11) 
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C-planc 



-(7- 



g+g-4 



)V = T- 00 





C = -V^T( S + £± | 











oo 




-V=i - 





2+^)^1 + 00 



Figure 4.5.2 

Now, we recall from Section [4~3l that §' is the dual space ofS = ^(SQR)) 
via the following diagram: 



S' 

u 



U 

s 



± r p+q-5 



dr) 



S'(R) 

U 

L 2 {R) 

U 
S(M). 



Diagram 4.5.3 

We are ready to prove the main result of this section: 

Proposition 4.5.6. 1) Ki k belongs to §' . That is, a-Ki^ £ §'(R). 

2) The Fourier transform 3^(a^Ki k)(C) of °~-Ki,k ^ s equal to -^=i/j(() 
(see (|4.5.5p for definition). In particular, \3^(a^Ki^)(C)\ = ^ f or C £ ^- 



Proof. It follows from Lemma 14.5.41 (1) that ijj is a tempered distribution, 
and therefore, its inverse Fourier transform < J~ lr 4> E S'(R). We also know 
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that a-K^k G C°°(M) by definition. Let D'(R) be the space of distributions, 
namely, the dual space of Cq°(R). In light of the inclusion 

S'(M) C D'(R) D C°°{R), 

all the statements of Proposition 4.5.6 will be proved if we show 

y/ira-K ltk = 3""V in D'(M), (4.5.12) 

that is, 

tt (a-K ltk )(x)$(x)dx = {T-^){x)^p)dx (4.5.13) 



holds for any test function <j> G Co°(R). In fact, (|4.5.12p will imply that 
a-Ki k G S'(M) and y/n$(a_Ki t k) = ip as a tempered distribution. 

The key to the proof of (4.5.13) is the integral expression of Kn-(t) stated 
in Lemma 14.5.51 By ()4.5.1ip . the left-hand side of ( 4. 5.13 ) amounts to 



1 



2tt 
1 



^(C)e _v/3Tx ' C ^C )<t>{x)dx 



2vr J L 



V 

oo 



^(CX^XCK 

= right-hand side of (|4.5.13h . 

In what follows, we explain the above equalities in details: 

First equality is by Fubini's theorem. For this justification, take a > 
such that Supp</> C [—a, a]. Then, 



< \\6\\„e aT > 



[x)e » ~">\ s \m\ooe- 
for C = C — V~^V with r\ > 0. Here, ||0||oo denotes the L°° norm. 

2 

m<v< m 

for some constants rji and 772 such that rji > \ if |£| is sufficiently large. 



Since 7 > — p+ i 5 , we may assume £ = £ — \/—lr] G L' satisfies 



Then, there exists a constant C > such that |t/>(C)| — 2?? as l£l — * °°> 
by Lemma 14.5.41 (3). Thus, if |£| is sufficiently large, we have 



moH^ v ~ lxC \<cu\u^. 
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Hence, ^(O^l^) 6 " is absolutely integrable on L' x [—a, a] (and there- 

fore, on V x (—00, 00)). Thus, we can apply Fubini's theorem. 

Second equality follows immediately from the definition (|1.2.ip of the 
Fourier transform. 

Third equality follows from Cauchy's integral formula. First, we ob- 
serve that -(/>(£) (3~0)(£) is holomorphic in the domain between the two inte- 
gral paths (—00, 00) and L' since its poles lie only on {— \J — lm : m = 1, 2, • • •} 
(see Lemma 14.5.4 (2)) . 

Next, let us show 

rvo 

lim / \^-^l v )(^m + V^lv)\dv = (4.5.14) 

+00 J 

for a fixed r] (> 7 + E± ^) (see Figure \4J>M ■ T « see (I4.5.14jl . take a > 
as before such that Supp</> C [—a, a]. Then, by the Paley- Wiener theorem, 
there exists a constant C such that \$4>($, + y/—lr])\ < Ce ari . Now combining 
this with Lemma 14.5.41 (3). we get 

momT)\ < c\ar 2v asiei-oo 

for C = C — V — 177 and bounded r\. Hence, (|4.5.14p is proved. By Cauchy's 
integral formula, we get the third equality. 

Last equality is by the definition of the Fourier transform for tempered 
distributions: 

(f,g) = (^f^g) f e S'(R), 9 e §(R). 

Hence, we have proved f)4.5. 12fl . Now, the proof of Proposition 14.5.61 is 
completed. □ 

4.6 Proof of Theorem HXTI (2) 

In this section, we complete the proof of Theorem 14.1.11 (2). For this, it is 
sufficient to show the following proposition: 

Proposition 4.6.1. K\ & = Kik- 

Here, we recall that the kernel distribution ki & of TJ & is given in Propo- 
sition H33I and that Ki jk is defined in (|4.1.2|) . 

Proof. The proof makes use of the following: 
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Lemma 4.6.2. Let k±,K2 G If there exists <j) G 8 suc/i i/iai 



y((7 + 0)(C)^O /or any CGM, 



(4.6.1) 
(4.6.2) 



then K\ = K2- Here, we recall from (4.3.9) for the definition of A, 



K 



Proof of Lemma \4-6.2\ The identity (|4.6.2p implies 



CT_Ki * <7+(/> = <T_ft2 * 0~+4> 



by the formula (4.3.10) of A K . Therefore, we have an identity 



in S'(IR) by taking their Fourier transforms. 

On the other hand, it follows from the assumption that o+0 G §(R) and 
its Fourier transform 9 r (o"+0) does not vanish on K, we can divide the above 
identity by 9 r (cr+0)(^), and obtain the following identity in S'(R): 



We want to apply Lemma T4.6.2I with k\ := k^, K2 := Ki,ki 4> '■= fl,k 
(see (|3.1.2 ) for the definition). The conditions in the lemma will be verified 
by the following steps. 

Step 1. Ki t k, Ki t k G 8'. These statements have been already proved in 
Propositions 14.4.41 and 14.5.61 

Step 2. fi y k G 8. This has been proved in Lemma 14.4.1 i 

Step 3. 9"(<T + // i fc)(C) has no zero points on R. This statement will 
follow readily from Claim 14.6.31 We note that we have assumed p > q > 2 
and p + q > 6. 

Claim 4.6.3. We recall from (|3.1.1|) the definitions of Cases 1 and 2. Then, 



y(<7_«l)(C)=3 r (<7_K 2 )(C). 



Hence, <t_ki = <t_K2, and in turn, ki = K2- 



□ 





Case i, 
Case 2. 



(4.6.3) 
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Proof. By the definition (I3.1.2|) of fi tk and the definition (|4.3.3p of a + , we 
have 

M lk) ( X) = 4= e (^-» x [ K :^ 2ex) case 

+ ' A 7 V2 l^+^e*) Case 2. 

In Case 1, we have 

= _L ^ \t l + k + P ^-Z+^K q _^ +k (2r)dr. 

Applying the formula (|7.2.1ip of the Mellin transform for the .ff-Bessel 
function, we obtain the right-hand side of ( 14.6.30 . Likewise, in Case 2, 
3(o-+fi,k)(C) is equal to 



J —oo 2 



Switching the role of (p, I) and (q, k), we see (|4.6.3p holds also in Case 2. □ 
Step 4. A Kl k fi jk = A Kl k fi >k . 

To see this, we prepare the following explicit formulas. As we shall see 
below, the proof of (1) is algebraic by using the fact that tt(wq) acts on each 
if-type as ± id. On the other hand, the proof of (2) is based on an explicit 
integral computation. 

Claim 4.6.4. Let a(l,k) be as in (|3.1.5p . 

1) A Kl j l>k = (-ir^wA fe . 

2) A Kl J, k = (-1)^)+^/,, 



Proof of Claim \4-6.4\ 1) The function fi tk belongs to the i^'-invariant sub- 
space H^ k (see (|3.1.4p ). and therefore, by Theorem 13.1.11 (3). we have 

n(w )(f ljk W) = (-l) o(, ' fc)+£ ^/ {)fc # 

for cp G ^(Rf- 1 ) and tp G Ji k (R q ~ l ). In light of the definition of T l>k (see 
Theorem 14.1.11 (1)). this implies 

Ti, k fi, k = (-l) a{l ' k)+E ^fi, k - 
By the definition of A Kl . (see Proposition 1 4.4.4"]) . the first statement follows. 



9G 



2) First, we treat Case 1, namely, the case + I — k > 0. Then, 



1 / P+q-i 

r 

V 



8vr y L , V 4 
( " 1)/+£? ^rf ; + fe ->r g + 3 ^" fc " g )^ (4.6.4) 



SvrV^T 7l V 2 



2 y ' U "V 2 2 

= (-l) a{l ' k)+E ^fl,k(r)- 

Let us explain the above equalities in more details. 

First equality. This is by the integral expression of Ki k (see Lemma 
Ha and the definition of A Kl h . 

Second equality. We recall the upper estimate of IVKOI given in 
Lemma 14.5.41 (3) and the asymptotic behavior of fi,k{ r ') ( see (|3.4.1|) and 
(13X2"]) ). Then, in light of 

mOr'^+^AkWl < mC)\r ,E± ^ +v \fl,k(r')\ for C = £ - V^Ir,, 

the second equality follows from Fubini's theorem. 
Third equality is by Lemma 13.4.31 

Fourth equality is from the change of the variable as before: s = 



2 

Fifth equality. The poles of the integrand in (|4.6.4|) are of the form 
I + k + 2a (a € N) or -q + 3 + I - k + 2a (a 6 N). These lie on the right of 
the contour L because of the assumption + 1 — k > 0. Hence, the fifth 
equality holds by the integral expression of Meijer's G-function (see (|7.6.2|) 
in Appendix). 

Sixth equality follows from the reduction formula of the G-function 
(see (|7.6.13|) ). 

Seventh equality is by the definition (|3.1.2D of fik and the definition 
(pTT3]) of a(l,k). 
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Case 2 can be treated in the same manner. In this case, the integral 

l+k-s -p+3-l+k- s 
r ( g ) ( 2 ' 

arises instead of (4.6.4). But again, by the assumption + / — k < 0, 
this defines the G-function which reduces to //^ by the same reduction 
formula. □ 

Step 5. K[,k = Ki jk . 

Steps 3 and 4 imply k^j- = Ki t k by Lemma 14.6.21 Thus, Proposition l4.5.6l 
is proved. □ 

Now the proof of Theorem 14. 1. 1 1 finishes . 
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Chapter 5 

Main theorem 



This chapter is a highlight of this book. We find an explicit formula for 
the 'Fourier transform' "Sc ° n the isotropic cone, in other words, we find an 
integral kernel for the unitary inversion operator tt(wo) on the Schrodinger 
model L 2 {C) of the minimal representation. The main result is Theorem 



5.1 Result of this chapter 

Let C be the conical variety {x 6 W +q - 2 \ {0} : Q(x) = 0} where Q(x) = 
x\ + ■ ■ ■ + x 2 _ l — x 2 — ■ ■ ■ — x 2 + 2 - We recall from the Introduction that the 
generalized function K{x, x') on C x C is defined by the following formula: 

K{x, x) = K(p, q; x, x) := c p>g $ p> q((x, x'}), (5.1.1) 

where (•, •) stands for the standard (positive definite) inner product on 
W +q ~ 2 . The constant c v , q and the Bessel distribution $ P)? (t) are given 
as follows (see (ITXT]) . (fTXT l. and ( DT5j) ): 



2( _ i) a^ E±a 

7T 2 



(5.1.2) 



(t) if min(p,g) = 2, 
*P,<z(*) '■= { ^ p+|-e (*) if p, <? > 2 are both even, (5.1.3) 
(i) if p, (/ > 2 are both odd. 



We are ready to state the explicit formula for the unitary inversion op- 
erator: 
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Theorem 5.1.1 (Integral formula for the unitary inversion operator). 

Let (ir,L 2 (C)) be the Schrddinger model of the minimal representation of 



G = 0(p,q) forp,q > 2 andp+q > 6 even, andwQ = 




. Then the 



unitary operator tt(wo) : L 2 {C) — > L 2 {C) is given by the following integro- 
differential operator: 

%(w )u(x) = I K(x,x')u(x')dfi(x'), ueL 2 (C). 
Jc 

The right-hand side of (|5.1.4p involves a singular integral. It factors through 
the Radon transform (see Section 5.2), and we shall see that the right-hand 
side of (|5.1.4B is well-defined for any compactly supported smooth function 
u on C, and extends as a unitary operator on L 2 {C). 

As for the Bessel distribution & Pt q, we shall give a Mellin-Barnes type 
integral formula for in Section 6.2, and the differential equation satisfied 
by $ Pi9 in Section [B31 

Since the action of the maximal parabolic subgroup P max on L 2 (C) is of 
a simple form (see f)2.3. 10[) — f|2.3. 13 j) ) . Theorem 15.1.11 gives an explicit action 
of the whole group G on L 2 (C) because G = P max ]J P max io P max . 

Theorem 15.1.11 immediately yields two corollaries about the Plancherel 
formula and the reciprocal formula of our integral transform. 

Corollary 5.1.2 (Plancherel formula). Let 3 C : L 2 (C) L 2 (C) be an 
integral transform against the kernel K(x,x') (see (|5.1.1|) ). namely, 

($ C u)(x):= [ K(x,x')u(x')dfi(x'). (5.1.4) 
Jc 

Then Jq is unitary: 

II 1 1,2 ( C ) = ||«|| i 2( C .). 

Since the group law w 2 = 1 in 0(p, q) implies 7r(w;o) 2 = id on L 2 (C), we 
immediately obtain the following corollary to Theorem l5.1-H which can also 
be viewed as giving the inversion formula J"^ 1 = 3^c- 

Corollary 5.1.3 (Reciprocal formula). Retain the notation as in Corollary 
\5.1.2i The unitary operator $c is of order two in L 2 (C). Namely, we have 
the following reciprocal relation: 

(3^cu)(x) = / K(x,x')u(x')dfj,(x'), 
Jc 

u{x) = I K(x,x')($cu)(x')d[i(x'). 
Jc 
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Remark 5.1.4 (Comparison with the Schrodinger model of the Weil rep- 
resentation, see Section 1 1 .6[t . In the case of the Schrodinger model of the 
Segal-Shale-Weil representation w of the metaplectic group Mp(n,l), the 

corresponding 'inversion' element w acts on L (M n ) as e * 9tr», where 
9"ign denotes the Fourier transform. We note that (w'q) 4 gives the unique 
non-trivial element £o in the kernel of the metaplectic covering Mp(n, R) — » 
Sp(n,M), and ro(£o) = — id. This fact reflects the identity 5Fj| n = id on 
L 2 (M n ). Thus, the above two corollaries can be interpreted as the counter- 
parts to the Plancherel formula and the equality = id of the Fourier 
transform CT^n on W 1 . 

Remark 5.1.5. In 14 6[ Corollary 6.3.1], we gave a different proof of the 
same Plancherel and reciprocal formulas in the case q = 2 based on analytic 
continuation of holomorphic semigroup of operators. 

This chapter is organized as follows. In Section 15.21 we analyze the in- 
tegral transform (15.1.40 by means of the (singular) Radon transform. In 
particular, we prove that the integral transform (|5.1.4p is well-defined for 
u £ C^°(C) in the sense of distributions. The second step of the proof of 
Theorem 15.1.11 is to use the restriction from G to K ' = K H M™ ax (see 
Section l2.3p and is to show the (l,k)-th spectrum of the unitary inversion 
operator S'c coincides with the radial part 2} & of Jc when restricted to 
each if'-isotypic component ^(W' 1 ) (g> JC fc (M 9_1 ) (see Lemma l5XTj) . The 
latter operator 7) ^ was studied in details in the previous chapter (see The- 
orem I4.1.ip . Section 15.31 explains a general formula giving spectra of a 
-RT'-intertwining integral operator. In Section 15.4} we reduce the proof of 
Theorem 15 . 1 . 1 1 to the identity of spectra for specific if'-intertwining integral 
operators. This identity is verified in Sections 15.51 and 15.61 Some technical 
parts (e.g. Mellin-Barnes type integral formulas of Bessel distributions) will 
be postponed until Chapters M and H 

5.2 Radon transform for the isotropic cone C 

This section studies the kernel K(x,x'). The subtle point in defining K(x, x') 
consists of the following two facts: 

1) The distribution $ P]? (t) is not locally integrable near t = 0. 

2) The level set (x,x') = t is not a regular submanifold in C x C if t = 0. 
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In fact, the distribution $ P)9 (t) involves a linear combination of distributions 
5( k -V(t) and t~ k {k = 1, 2, . . . , E± ^) as we shall see in Section EH on the 
one hand, and the differential form 

dQ(x) A dQ{x) A d((x, x) - t) 

of x, x' vanishes if (x, x') belongs to the submanifold 

Y := {{x,x') € C X C : Rx = R(I p>q x')} 

on the other hand. Here, we note that Y C {(x,x') : {x,x') = 0}. 

Our idea to give a rigorous definition of K(x, x') is to factorize the trans- 
form (15.1.4P by using the singular Radon transform and then to find a Paley- 
Winer type theorem of the singular Radon transform. 

Let 6 denote the Dirac delta function of one variable. The Radon trans- 
form of a function if on R p+q - 2 is defined by the formula (see for example 
[251 Chapter I]): 

(Rip)(x,t):= ip(x')5(t - (x,x'))dx' , (5.2.1) 

for x £ RP+i~ 2 \ {0}, f S M. 

The Radon transform Rip is well-defined, for example, for a compactly 
supported continuous function tp. More generally, Rip makes sense if tp 
is a compactly supported distribution such that the multiplication of two 
distributions <p(x') and S(t — (x,x')) makes sense. 

Now we recall the injective map (see (2.2.6)) 

T : L 2 (C) -> S'iRP+i- 2 ), u^u5(Q) 

yields a compactly supported distribution Tu if u G Cg°(C). In this context, 
what we need here is the following result: 

Let Cq(M) denote the space of compactly supported functions on WL with 
continuous derivatives up to k. 

Lemma 5.2.1. Suppose u £ Cq°(C). 

0) The Radon transform Jlu(x,t) := R(Tu)(x,t) is well-defined and con- 
tinuous as a function of(x,t) ECx (R\{0}). Moreover, there exists A > 
such that 

Suppftu C {(x,t) E C x (M\{0» : t < A\x\}, 

where \x\ := (x 2 + • ■ ■ + x 2 +fJ _ 2 )2 . In particular, Jlu(x,t) vanishes if \t\ is 
sufficiently large for a fixed x £ C. 
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1) If Pi Q > 2 and p + o > 8, i/ien i) extends continuously to t = 
and ttu(x, •) G C#(K) w/iere fc := 

2) 7/min(p, g) = 2, i/ien Ru(x,t) is bounded on C x (R \ {0}). 

3) Ifp,q>2 and p + q = 6 (namely, (p,q) = (3,3)), then there exists 
C = C(x) > siic/i tfiaf 

|i?(Tn)(x,t)| < C|log|t|| 

j/ 1 is sufficiently small. 

Proof. See IM]- □ 
We note that 

0iu(x,t) = / f(x')8((x, x 1 ) — t)dp,(x') 
Jc 

is well-defined for (x,t) G \ {0}) x (R \ {0}), but we need here only 

the case where x £ C. 

We rewrite the right-hand side of (|5.1.4p for u £ C^°(C) as follows: 

(3 C u)(x) := [ K(x,x')u(x')dn(x') 
Jc 

= c P ,q § p>q ((x,x'))Tu(x')dx' bv (|5.1.1|) 

= Cp,q / $p t q((x,x'))Tu(x')duj(x')dt 
JR J{x,x')=t 

= Cp >q I $ Piq (t)0lu(x,t)dt. (5.2.2) 

Lemma 5.2.2. T7ie right-hand side of (|5.2.2p is well-defined for u G Cg°(C). 

Proof. It follows from Theorem 16.2.11 which we shall prove later and from 
the definition ()5.1.3p of the distribution $ Ptq (t) that $> p<q (t) has the following 
decomposition: 

%, g (t) = %l(t) + ^(t), 
where $^|(t) and §> p ™ s (t) are distributions on R such that 

1) is a locally integrable function on R for any sufficiently 

small e > 0, 
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2 ) ( 

if min(p, q) = 2, 



P>9 ^ ' 



p+q-6 

? 2fc("m-fc)! ^ ^ P ' " > 2 b ° th 6Ven ' 



P+g— 6 

2 



- V -rr — Htt* - * if P, g > 2 both odd. 
vr f~" 2 k (m-k)\ 

k=l 



We note that &p^ s (t) / only if p, q > 2 and p + q > 8. Combining with 
Lemma I5.2.11 we see that the right-hand side of (5.2.2) is well-defined in all 
the cases. □ 

Lemma 5.2.2 defines a linear map 

7 C : C^{C) -» C°°(C), 

and defines K{x,x') as a distribution on the direct product manifold C x C. 
In Section [5^1 we shall see that the image 9 r c , (Cg°(C)) is contained in L 2 (C), 
and extends to a unitary operator on L 2 (C), which in turn equals the 
unitary inversion operator tt(wo). 

Remark 5.2.3. The plane wave expansion gives a decomposition of the Eu- 
clidean Fourier transform on L 2 (M n ) into the one- dimensional integral 
transform (Mellin transform) and the Radon transform, namely: 

(J R nu)(x) = c n {^,(Ru)(x,-)), 

— / 

where c n := (^) 2 and V(t) := e v_1 * (e.g. JM Chapter I, §1.2]). In this 
sense, the formula (|5.2.2p can be regarded as an analog of the plane wave 
expansion for the unitary operator tt(wq) on L 2 (C). 

5.3 Spectra of i^'-invariant operators on S p ~ 2 x S q ~ 2 

The expansion into spherical harmonics 

oo 

L 2 (S n ~ l ) ~ YT ^( Rn ) 

1=0 

gives a multiplicity- free decomposition of 0(n) into its irreducible represen- 
tations (see Appendix I7.5p . and consequently, any 0(n)-intertwining opera- 
tor on L 2 (S n ~ l ) acts on !K l (M. n ) as a scalar multiplication owing to Schur's 
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lemma. The scalar is given by the Funk-Hecke formula (see [1] §9.7], see 
also |46l Lemma 5.5.1]): for an integrable function h on the interval [—1, 1] 
and for <fi G !K'(IR n ), 

/ h((u,u'))(/)(u')du' = ci tTl {h)4>{oj), 

where the eigenvalue Q jn (/i) is given by 

on-2 — M /■! 3 

«>M = T(n-2 + l) J ! Kx)Cl 2 " x ^~ dx - ( 5 - 3 - 1 ) 

Here, C^{x) stands for the normalized Gegenbauer polynomial (see Section 

E3D. 

Likewise, any /^'-intertwining operator on L 2 (S P ~ 2 x S q ~ 2 ) acts on the 
subspace JC Z (M P_1 ) <g> J{ fc (]R 9_1 ) as a scalar multiplication for each k, I G N 
(we recall X' ~ 0(p — 1) x 0((/ — 1)). In this section, we determine this 
scalar for specific intertwining (integral) operators. In particular, the scalar 
in Example 15.3.21 will be used in the proof of our main theorem (Theorem 
ECU). 

We begin with a general setup for a iC'-intertwining operator on L 2 (S P ~ 2 x 
S q ~ 2 ). Let h be an integrable function of two variables on [—1,1] x [—1,1]. 
We consider the following integral transform: 

B h : C{S P ~ 2 x S q ~ 2 ) -► C{S P ~ 2 x S q ~ 2 ), 

ip(oj,r)) > / h((u, a/), (rj, r]'))(p(oj', T])(kJ dr\ . (5.3.2) 

Jsp- 2 xSi- 2 

Lemma 5.3.1. B h acts on each K' -type ^(W^ 1 ) (g> ^(R 9-1 ) by a scalar 
multiplication of a^k(h) G C. The spectrum ai t k(h) is given by the following 
formulas. 

1) 7/min(p, q) = 2, say q = 2, then for k = 0, 1, 

<*M = r(p- 3 + o / MWCi 2 " x )~ dx > ( 5 - 3 - 3 ) 

where we set 

(U k h)(x) := h(x, 1) + {-lfh{x, -1). (5.3.4) 
For fc > 2, Qi{ )fc (/i) = 0. 
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2) Ifp,q> 2, then 



~ r(p - 3 + l)T(q - 3 + k) 

[ h(x,y)C^(x)C^(y){l-x 2 ) E ^ ± (l-y 2 )^ ± dxdy. (5.3.5) 
l J -i 

Proof. 1) If q = 2, then S^ 1 x S^ 1 = S^ 1 ]} S 7 '" 1 (disjoint union), and 
Ji k (M 1 ? -1 ) = if fc > 2 (see Section [73]). Then, the formula (|5X3|) is 
essentially the Funk-Hecke formula (5.32]) for 

2) Applying (|5.3.ip to each factor, we get (|5.3.5|) . □ 

Let us give some examples of the explicit computation of spectra a^k(h). 

Example 5.3.2 (Riesz potential). Consider the following Riesz potential 
for Re A > -1: 



h ± ( v (a + y)± 
K&v) : =TxaTT)' 

(x + y) A 



*/ e(s + y) > 0, 



r(A + l) v ' ' (5.3.6) 

if e(x + y) < 0, 

where e = ±1. Then, the spectrum ai t k(h x ) for the K' -intertwining operator 

2l _A 7r £± F l (±1)i+fcr(A + £± p4 ) 



B h ± amounts to 

A 



2 

( ±1 )J+k 



— r^A + J sin^ — 2"^tt J sm^-^-^— r -vrj7i ifc (A), 

where we set 



r(^A)r(^±3+b^) 

j-^ A+p+g+^+fc-4 -^p^ A+p-l+Z^fc - 



7/ifc (A):=2 vr 2 ,,,_J,,, fe =t— " ■ ■ (5.3.7) 



Proof of Example 1 5. 3. i?l Use (|5.3.5p . We postpone the actual computation 
of the integral (the first equation of a^k(h^)) until Appendix (see Lemma 
17.9.11 with fj, = ^y^jV = ^^). In the second equation of ai t k(h^), we have 
used the functional equation r(z)r(l — z) = sin J zw ^ ■ □ 
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We define a kernel function h\(x,y) = h p ^ q (x,y) with parameter A as 
follows: 

{{x + y)\ if p, q > 2 both even, 

f^±yA + + itpj9>2bothodd . 
\ tanA7T smA7r / 

(5.3.8) 

Proposition 5.3.3. Let ReA > —1. For a kernel function h\ (see (|5.3.8p ), 
the spectrum cei t k(h\) given in Lemma \5.3.1\ amounts to 

ai,k(h\) = ^ — x+p+q+l+tc _ 4 A+p „ 1+ /„ fc (5.3.9) 

_ (jgW r(^)r( -^-^- A ) 

Proof. The second equation (5.3.10) follows from the identity (4.5.3]) of 
gamma functions. Let us show the first equation (|5.3.9j) . In terms of h/jr 
defined in ([513. 6), we rewrite h\ (see (j5.3.8j) ) as 

■ if p, q both even, 



r(A + 2±pi)sin(-Avr) >4 + if p, g both odd. 

V tan(A7r) sin(A7r) ^' ^ 

(5.3.11) 

Since a^ k is linear, i.e., ai jk (ah + bg) = aa^ k (h) + bcti jk (g), o,kC, by 
(|5.3.1ip . we have 

ai,k(h\) = C/fc(A)7; fe (A), 



where 7zfc(A) is the meromorphic function given by (|5.3.7p . and 
C ^( A ) := / > x x 



n A -'- fc+2 7rsin A+g -^- t+ * 7r Jl if p, 9 both even, 

sin ("^) Issfe + fe^i? if P, 9 both odd. 



Hence, the proof of Proposition will be completed by the following claim: □ 
Claim 5.3.4. 

Ci,*(A) = ^ • 
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Proof. Let us first consider the case where both p and q are even. Then, the 
two integers — I — k + 2 and q — 1 — I + k have different parities. Hence, 

„ , , 9-2 sin ^7T cos (— 1)' +£ 2~ 
C, )fc (A = -1 2 2 = • 

sm ATT 2 
Next, suppose both p and g are odd. Then, 

1 I ( 7 l)' +fc ft^' 



tan(Avr) sin(Avr) [-tan^vr, 

sin sin = izil^tan^, 

9 + 1 



sin(-A)7r " I / ir ;+2±l cos^7r = (-1)-'+ 



tan fn ' 



according as I + /c is even (upper row) and odd (lower row). Thus we have 



C i)fc (A) 



(-1) 



9-1 
2 



•i 



2 

in either case. Hence, Claim [5~.3.4I is verified. □ 
Let T be the triangular domain in M 2 given by 

T := {(x, y) G M 2 : x < 1, y < 1, < x + y}, 
and define a function g\(x,y) with parameter A G C by 

J(x + y) A (l-x 2 ) £ ? i (l-y 2 ) 2 2 1 (x,y)eT, 

Lemma 5.3.5. 1) For Re A > —1, g\ is a distribution of compact support, 
and with holomorphic parameter A. That is, {g\, 99) is holomorphic in {A € 
C : ReA > -1} for any ip G C°°(IR 2 ). 

2) 0^ extends as a distribution with a meromorphic parameter A G C. 
77ia£ is, (y,\ , ¥>) is a meromorphic function with respect to A G C /or any 
99 G C°°(IR 2 ). 

Proof. The first statement is clear because y^ £ F 1 (M 2 ) if ReA > —1. For 
the second statement, we rewrite g\ as 

g x (x,y) = (x + y) x + (l - x)*f (1 + x)f {1 - y) 3 ^ (1 +y) 3 f. 
Then, Lemma follows from Bernstein's theorem [5]. □ 
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5.4 Proof of Theorem 15.1.11 

We recall from (|5.1.4p that "5c is the linear map denned by 

Jc 

Since 3~c commutes with the if'-action (K' ~ 0(p — 1) x 0(q — 1)), $c 
preserves each i^'-isotypic component of L 2 (C) given in the decomposition 
(see gXQ): 

oo 

L 2 {C) ~ J]® L 2 (R + , -r p+(1 ^dr) ® ^(M^ 1 ) 8) ^C fe (R 9_1 ). 

l,k=0 

On the other hand, we have seen in Theorem 14. 1. 1 1 that n(wo) also preserves 
each if'-isotypic component, and accordingly has a decomposition: 

oo 

tt(wo) = ® id ig) id, 

i,fc=0 

where 2]^ is a unitary operator on L 2 (R + , ^r p+<7_5 dr) whose kernel K[^{t) 
is explicitly given in (14.1.2p . 

We shall show the equality 3"c = tt(wo) by restricting to each (I, k) 
component, namely, 

Lemma 5.4.1. For each l,k 6 N, we /icroe 

3 r c|L2(R + ,rP+8- 5 dr)ig)?<; ! (KP- 1 )®J<: fc (K8- 1 ) = <8> id <g> id . (5.4.1) 

Instead of proving Lemma 15.4.11 we shall prove Lemma 15.4.21 on the 
spectra ol\ & and the kernel functions Ki^, which turns out to be equivalent 
to Lemma 15.4.11 For that purpose, we set 

h r y(x,y) := Cp !q $ Pt q(rr'(x + y)), (5.4.2) 

where c Pj(? and <3? p <q are defined in ()5.1.2p and (|5.1.3p . Then, by the definition 
(|5.1.ip of K(x,x'), we have 

= h r y((uj,Uj'}, (??,?/)). 
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Suppose f(r)u{u),rj) £ L 2 {R + , r P +q ~ 5 dr) g> ^{W" 1 ) <g) Jf fc ( ffi ^ 1 )- 
= J c K^ r ^,x^(fu){x'W(x') 

= \ I I I h r y((u,uj'),(ri,ri'))f(r')u(u',r]')r' p+q ~ 5 dr'duj'dr]' 
2 Jo Jsp~ 2 J si- 2 
1 

~~ 2 



oo 

( J B Vi ,u)/(r / )r / ^ 5 dr / . 
Since ,u = ai^(h r y)u by Lemma 15.3.14 we have 



o 



3 C {fu){ru, r V ) = I I aitk (h r y)f(r'y p+ i- 5 dr'u(uj, V ). 



o 



2 

On the other hand, it follows from Theorem 14.1.11 (2) that 



((T hk ®id®id)(fu))(ru,rri) 
= (Ti,kf)(r)u(u,ri) 
1 



2 



K lik (rr')f(r')r' p+q - 5 dr'u(io, rj). 



o 



Since L 2 (R + , r p+ i- b dr) <g> ^(W 1 ) <g> 5£*(R9 _1 ) is spanned by a linear com- 
bination of the function of the form / {r)u{u , ij) , Lemma 15.4.11 is equivalent 
to the following formula between kernel functions: 

Lemma 5.4.2. For each l,k 6 N, we have 

Ul,k(K,r') = K iM (rr'). 

The proof of Lemma 15.4.21 will be given in the following two Sections, 
which will then complete the proof of Theorem 15.1.11 

5.5 Proof of Lemma 5.4.2 (Hermitian case q = 2) 

This section gives a proof of Lemma f5.4.2l in the case min(p, q) = 2. Without 
loss of generality, we may and do assume q = 2. By the definition (|5.1.3p of 
&p2(t)i the definition (5.4.2) of h r y amounts to: 

h r y(x,y) = c p ,2$t-4 (rr'(x + y)). 
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Since <3?~p~_ 4 (t) is a locally integrable function supported on t > (see 

~2~ 

Theorem I6.2.ip . we have from the definition (|5.3.4p of the operator Uk (k = 
0,1): for -1 < x < 1, 

(C/feV)(z) = Cp.s^ti^r^x + 1)) + (-l) fc c P)2 $^4(r-r'(x - 1)) 

2 2 

= c p , 2 $t4(rr / ( 2; + l)). 

2 

Then, by (|1.3.ip . we have: 

(p-l)(p+2) 

2 4 ( — 1) 2 p-4 p-4 



(rr / )"~(x + 1)" * J s= ±(2y/2rr J ^c + 1)). 



P-2 

— 

Applying the Funk-Hecke formula ()5.3.3p . the spectrum ai t k(h r y) amounts 
to 

a «M= vh-(p-3 + o (rr) 4 x 

Jp-4 (2 v /2rr / (x + l))cf^(x)(l + x) E * 1 (l -x^cfe 
■ l 2 

(p-l)(p+2) | . „ p-3 



= 4(-l) a +l (rr')- — Jp-3+2l(Wrr>) 

= {-l)^K ljk (rr'). 

Here, the second equality follows from the integral formula (|7.4.10p of the 
Gegenbauer polynomials in Appendix with a = 2y/2rP and v = and 
the last equality follows from (|4.1.4|) . Since p is even in the case q = 2, 
the right-hand side is equal to Ki^{rr'). Hence, Lemma 5.4.2 is proved for 
q = 2. □ 



5.6 Proof of Lemma 5.4.21 (p, q > 2) 

This section gives a proof of Lemma 15.4.21 in the general case p, q > 2. 

First, we give an integral formula of Mellin-Barnes type for h r y(x,y) 
(see (|5.4.2p ) by means of h\(x,y) (see (|5.3.8p for definition): Suppose p,q > 
2. 

Claim 5.6.1. Let 7 > —1 and L be a contour that starts at 7 — \J — loo and 
ends at 7 + V - loo. We assume that L passes the real axis in the interval 
(~ P+ 2~ 4 ' ~ P+ 2~ 6 )- Then, we have 

h r y(x,y) = [ (2rr') x h x (x,y)dX. (5.6.1) 

2tt\/ — 1 Jl 
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Proof. By the definition (|5.3.8p of h\(x,y) = h p ^ q (x,y) and the integral 

formulas ([6X2]) and (^2^j) of ^ + +g _ 6 (t) and * p-j-q — 6 (t) respectively, we 

-a — 2 

have 



1 



s x h\(x,y)d\ 



p +g _ 6 (- L ^ Z ) p,geven, 



q odd, 



for s > 0. In either case, it follows from the definition (|5.1.3p of that 

's(x + yy 



2tt 



i= J^s x h x (x,y)dX = 



Hence, we get Claim 5.6.1 by the definition (15.4.2ft of h r y(x,y). 
By the linearity of a^, we have 



(5.6.2) 



□ 



a l,k(K,r') 



a^ k {h x ){2rr') x d\ 



LJl 1_ / 1^ 2 r_V 2 

^^/rx y L Y( x+ i> + <i +l+k - 4 )T( x+p ~ 1+l ~ k ) 

Ki 7k (rr'). 



Here, in the second equality, we applied Proposition 15.3.31 and then used 
the equality (— l)' + [ a 2~l (— 1) 2 = (— l)' 4 "^, which follows from the 
congruence equality: 



(p-l)(p + 2) 



+ 



1 



p-q 



mod 2 



under the assumption that p + q is even. The last equality follows from 
Lemma 14.5.21 Hence, we have proved Lemma 15.4.21 in the general case 
p,q>2. □ 

Proof of Theorem \5.1.1l Now, Lemma f5.4.2l is proved in all the cases. Hence, 
the proof of Theorem 15.1.11 is completed. □ 
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Chapter 6 

Bessel distributions 



We have seen in the previous chapter (see Theorem 15.1. 1H that the unitary 
inversion operator ir(wo) : L 2 (C) — > L 2 (C) is given by the distribution kernel 
K(x,x') which is the composition of the restriction of the bilinear map 

CxC^I, {x,x')^ (x,x'} 

and Bessel distributions (see (jl.3.1jl — (fl.3.3)) of one variable. In this chapter, 
we analyze the distribution kernel from three viewpoints: integral formulas, 
power series expansion (including distributions such as 5^ k \x) and x~ k ), 
and differential equations. 

Section 6.3 gives a heuristic account on why K(x, x') is essentially of one 
variable, and why the Bessel distribution arises in K(x,x'). The results of 
Section [6731 is not used for other sections. 

6.1 Meijer's G-distributions 

In this section, we give a definition of Meijer's G- distributions which have 
the following two properties: 

1) They are distributions on K. 

2) The restrictions to the positive half line {x > 0} are (usual) Meijer's 
G- functions (see Appendix I7.6p . 

The main result of this section is Proposition I6.1.2T 

Let m, n, p and q be integers with < m < q and < n < p. Sup- 
pose moreover that the complex numbers a\ , . . . , a p and b\ , . . . , b q fulfill the 
condition 

a,j - b k ^ 1, 2, 3, . . . (j = 1, . . . , n; k = 1, . . . , m). 



113 



This means that no pole of the gamma function T(bj — A) (J = 1, . . . ,m) 
coincides with any pole of T(l — + A) (k = 1, . . . , n). We set 

c :=m + ?i — , (6.1.1) 



Throughout this chapter, we assume c* > 0. If c* = then we also assume 

p — q<0 or p — g>Re/i. (6.1.3) 

It is easy to see that the condition (|6.1.3p allows us to find 7 £ 1 such that 

7 > — 1 and (q — p)j > Re fi. (6.1.4) 

Remark 6.1.1. The conditions (I6.1.3P and 7 > —1 will not be used when 
we define (usual) Meijer's G -function as an analytic function in x > (see 
(7.6.2)). They will be used in showing that Meijer's G- distribution G(x+)l 
given by the Mellin-Barnes type integral (|6.1.8p is a locally integrable func- 
tion on R if we take an appropriate contour L (see Proposition \6.lM (3)). 

We now take a contour L which starts at the point 7 — \J — loo and 
finishes at 7 + yf— Too. Later, we shall impose the following conditions on 
L: 

L does not go through any negative integer. (6.1.5) 

L leaves bj (1 < j < m) to the right, and aj — 1 (1 < j ' < n) 

to the left. (6.1.6) 

We note that the condition (]6.1.6p implies: 
L does not go through any point in 

{bj + k : 1 < j < rn, k€ N}U{a,-l-fc : 1 < j < n, keN}. (jTTLBT 
With these parameters, we define a meromorphic function of A by 

m n 

nr(& i -A)nr(i-a i + A) 

ai,...,o p \ _ 3=1 j=i (6 17) 



r^(A 



b 1 ,...,b, 



n ni-bj+x) n rc^-A) 

j=m+l j=n+l 
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For Re A > —1, we set 



x x (x > 0) 
(x<0), 



(x > 0) 
|A (x < 0). 



•T 



Then, x^_ and x x are locally integrable functions of the variable x in M, and 
extend to distributions with meromorphic parameter A in the entire complex 
plane (see Appendix 7.1). 

Proposition 6.1.2. Let L be a contour satisfying (|6.1.5p and (|6.1.6p / . 
1) The Mellin-Barnes type integral: 



G(x+)i 



p,q 



2tt 



1 f 

/ -pm,n 



■ I 

A 



,bq 



L 

bx, 



,b q 



xid\ 



(6.1. 



is well-defined as a distribution on K. 
Its support is given by 

suppG(x + ) L = {x G R : x > 0}. 

2) If the contour L satisfies (j6.1.6j) . then the restriction of G(x + )l to 
the positive half line {x G M. : x > 0} is a real analytic function, which 

coincides with the (usual) G-function G^^ix ' ' p ) (see (|7.6.2|) for 

V bi,...,b q J 

definition). 

3) If the contour L is contained in the half plane {A G C : Re A > — 1}, 
then G{x + )l is a locally integrable function on R. More precisely, there 
exists eq > such that G(x+)l x^_ e is locally integrable for any e with < 
e < e . 



Likewise, we can define the distribution 
G{x^) L 



1 



ax, ■ ■ ■ , a p 
bx,...,b q 



2tta 



1 



ax, ■ ■ ■ , a. 
bx,...,b q 



p )x x _d\ 



by using the same contour L, and the support of G{x-)l is equal to the 
negative half line {i £ R : i < 0). 
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Remark 6.1.3. The distribution G(x±)l depends on the choice of the con- 
tour L even when we assume L satisfies the conditions (|6.1.5j) and (|6.1.6p . 
In fact, if L and L' are contours satisfying (|6.1.5p and (|6.1.6p . then G(x±)l 
may differ from G(x±)li by a distribution supported at 0, namely, a finite 
sum of Dirac 's delta function and its derivatives. This is because the distri- 
bution x± has simple poles at A = —1, —2, . . . , and consequently, its residues 
(see (\7.1.1\) and (|7.1.2|) ) may appear when we move the contour L across 
negative integers. In order to define the G -distribution in a unique fashion, 
we need to impose an additional constraint on the contour L. We shall work 
with concrete examples for this in Section [6.2\ where we use Cauchy's integral 
formula for distributions with meromorphic parameter. 



In order to prove Proposition 16.1.2} we need an asymptotic estimate of 
the T-factors in the integrand of (6.1.8) as follows: 

Lemma 6.1.4. For any e > 0, there exists a constant C > such that 



-pm,n I \ 



ai,...,a p 



< c e - wc * |ImA| |ImA| 



Re/z+(p— 9)7— 1+e 



for any A G L such that |ImA| is sufficiently large. Here, c* and fi are 
defined as in (j6.1.1j) and (|6.1.2p . and 7 = lim Re A. 

|Im A|^oo 



Proof. Fix a G C. By Stirling's asymptotic formula (4.5.8) of the gamma 
function, we have 



|r(o-A)| =C a |ImA| 
|r(l-a + A)| =<7 a |ImA| 



Re a— Re A— i 



Re a+Re AH 



s e -fl IniA l(l + 0(|ImAr 1 )), 

| e -||ImA| (1 + 0( | ImA |-l ))5 



as |Im A I tends to infinity with Re A bounded. Here, the constant C a is given 
by 

C a = v^e-f sgn(ImA)|Ima|_ 

ai , . . . , a p 
h,...,b q 

ing asymptotic behavior: 



By the definition (|6.1.7p of T 



y,q 



A 



we now get the follow- 



r-f[A 



ax,...,a p 
h,...,b q 



C' \lm\\ s e -f*l ImA l(l + OdlmAp 1 )), 
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as |ImA| tends to infinity, where C is a constant depending on Imaj and 
Imfej, and 



s = J2 Re (bj ~ A - -) + ^Re(-a, + A + -) 




£ Re(--6 i + A)- £ Re( % -A--) 




= Re fi + (p — q) Re A — 1, 
t = m + n — (q — m) — (p — n) 



= 2c*. 



As Re A converges to 7 when A G L goes to infinity, we get Lemma [6.1 .41 □ 

We are ready to give a proof of Proposition 6.1.2, 

Proof of Proposition \6.1.2L 3) We begin with the proof of the third state- 
ment. Suppose L is contained in the half plane {A G C : Re A > — 1}. We 
need to show the integral (|6.1.8l) makes sense and gives rise to a locally 
integrable function of x. For the convergence of the integral, we shall use 
Lemma 16.1.41 for the estimate as a function of A. The non-trivial part is a 
uniform estimate in the neighborhood of x = 0. Let us consider the interval 
< x < 1. 

Since the contour L has the property: 



the assumption L C {A £ C : ReA > -1} implies 5 > — 1, where we set 



7 = lim Re A > — 1, 

|Im A|— >oo 



5 := inf Re A. 



Hence, we get 

\xi\ <x 5 for < x < 1. 



On the other hand, it follows from Lemma 16.1.41 that 
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when |ImA| is sufficiently large. Here, we used the inequality Re/i + (p — 

q)j < (see (|6.1.4j) ) in the second case. Hence, T^^fx , P ) is 

V bi,...,b q / 

absolutely integrable on L in either case. Therefore, the integration (6.1.8) 
converges, giving rise to a function of x which is bounded by a scalar multiple 
of x s on the interval < x < 1, whence a locally integrable function of x. 
Thus, G{x + )l is locally integrable. Similarly, if we set 

e :=l + 8 (>0), 

then for any < e < eo, x~ e+5 is locally integrable, and consequently 
G(x + )l x+ e is locally integrable. Hence, the third statement of Proposition 
is proved. 

1) We divide the integral (|6.1.8|) into the sum of the following two inte- 
grals 



+ 

u Jc 



where L' is a contour contained in the right half plane {A £ C : Re A > — 1}, 
and C is the closed oriented curve given by L — V (see Figure I6.1.ip . 




Figure 6.1.1 
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Then, we have already seen that the second term gives a locally inte- 
grable function of x (the third statement of this proposition) . On the other 
hand, the third term is well-defined as a distribution because C is compact 
and the integrand is a distribution of x that depends continuously on A as 
far as A lies in C . Hence, the first statement is also proved. 

2) This statement is well-known. See Appendix 7.6 for details. □ 

6.2 Integral expression of Bessel distributions 

In this section, we apply general results on Meijer's G-distributions devel- 
oped in Section T6. II to special cases, and obtain the Mellin-Barnes type inte- 
gral expression for the distribution kernel of the unitary inversion operator 
tt(w ). 

Let m be a non- negative integer. We take a contour L such that 

1) L starts at 7 — \f— Too, passes the real axis at some point s, and ends 
at 7 + \/— loo. 

2) — 1 < 7 and — m — 1 < s < —m. 

Likewise, we take a contour Lq (with analogous notation) such that 

3) -1 < 70 and -1 < s < 0. 

For later purpose, we may and do take 7 = 70- See Figure [6.2.11 







I, 














7 




so 

4 — 1 


1 — h 

— m— 1 n 


— 1 

v — m 




H 

-1 








Figure 6.2.1 
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Then, we consider the following Mellin-Barnes type integrals: 



* iW! -5^r/ t r ( A + i + m) (at ^ tt - (6 ' 2 - 2) 

Mt) > f f J5>E + m\ dA , (6 ., 3 ) 



2tx\J — 1 T(A + 1 + m) \ tan(7rA) sin(-7rA) 

r j L r(. 



1 /" r(-A) / (2A)i (2t) A \ 
* m (t) := — -= / — \ . + dA. (6.2.4) 



We shall see that these integrals are special cases of (6.1.8) and define dis- 
tributions on R. The next theorem is the main result of this section, which 
will be derived from Proposition 16 . 1 . 21 by applying the reduction formula of 
Meijer's G-functions. 

Theorem 6.2.1. 1) 3^(i) and $ m (t) are locally integrable functions on M. 
Furthermore, for a sufficiently small e > 0, <&f n (t)\t\~ e and $> m (t)\t\~ € are 
also locally integrable. 
2) 

m / i \fe— l 

*„ + ,(t) = *, + „(t) - E 2 t (m _ t) , ^""w- («- 2 - 5 ) 
»„( t) = •„(*)- i Ej^r*-*- ( 6 - 2 - 6 > 

.S'ee (|7.1.5p m Appendix for the definition of the distribution t~~ k . In par- 
ticular, \P+ and ^ m are defined as functionals on the space C™~ 1 (R) of 
compactly supported functions on K wi/t continuous derivatives up to m— 1 
if m > 1 . 

The rest of this section is devoted to the proof of Theorem 16.2.11 
Before regarding the integrals (|6.2.ip -( |6.2.4p as those for distributions, 

we consider the classic cases, namely, their restrictions to M\{0}, which are 

real analytic functions. 

Let Li (i = 1,2,3) be contours that start at 7$ — \/— loo and end at 
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7i + v— Too, and pass the real axis at some point Sj. We assume 

st < 0, 

s 2 < — m, 
S3 < — m. 



¥ <7i, 
m 

T <72, 



Then, we have the following integral expressions of Bessel functions. Al- 
though the results are classical, we shall give a proof to illustrate the idea 
of passing from Bessel functions to Bessel distributions. The proof below is 
based on the integral expressions of Meijer's G-functions (see Appendix l7.61 
see also Proposition 16.1.21 (2)): 



Lemma 6.2.2. 1) For t > 0, 



J m (2V2t) = (2t)~ J m (2V2i) 



1 



r(-A) 



2) For t > 0, 

3) For t < 0, 

K m (2^2\t\) 



27r v / ^T7 Ll T(A + m+ 1) 
(2t)~Ty m (2v / 2i) 



(2t)id\. 



(6.2.7) 



1 



IX-A) (2t)j 
1 J L r(A + m + 1) tan(vrA) 



2tta 



(2|t|)--K m (2 v / 2|t|) 



dX. 



(6.2.J 



(-1) 



m+l 



r(-A) (2t)i 

4 v /r T A 3 r(A + m + 1) sin(vrA) 



dA. 



(6.2.9) 



Proof of Lemma 6. 2. 2. Each of the first equalities is by the definition of 
the normalized Bessel functions J m , Y m , and K m given in (|7.2.3j) . (|7.2.5|) . 
and (|7.2.6p . respectively. Let us verify the second equalities (the integral 
formulas for the Bessel functions). 

1) By the reduction formula (|7.6.12p of the G-function Gq2, we have 



(2t) _ T J m (2V2t) = G$(2t 



-m) 



for t > 0. Then, by the integral expression (|7.6.2p of the G-function Gq!], we 
have 



G$(2t I 0,-m) 



1 



2vn 



r(-A) 



lJ Ll T(l + m + A) 



(2t) x dX 
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for t > 0. Hence, (|6.2.7p is proved. 

2) By the reduction formula (|7.6.15p of the G-function Gf^, we have 



(2ty^Y m (2V2t) = Gf 3 [ 2t 



for t > 0. Then, by Example 7.6.2, we have 



— m — 2 
-m, 0, — m — 2 



^(20 
^13 



2/ 
1 



-m 



m, 0, — m —\j 

r(-m- A)r(-A) 



-(2t) x d\. 



2vr v / ^T 7 La r(m + f + A)r(-m — 5 — A) 
Now, (6.2.8) is deduced from this formula and the following identity: 
r(— m — A)r(A + l + m) 1 



r(m + § + A)r( — m 



A) tanwA 



for any m E Z. 



Here, the last identity is an elementary consequence of the formula T(z)T(l- 

Z) = -r*—. 

3) By the reduction formula (7.6.13) of the G-function Gg2, we have 

1 



{2\t\Y^K m (2^j2\t 



Gq® [ 2\t 



0, — m . 



Suppose t < 0. Then, again by the integral expression (|7.6.2p of Gq®, the 
right-hand side amounts to 



1 



4-7TA 



-1 



r(-A)T(-m- A)(2*)^dA. 



u>, 



Then, (6.2.9) follows from the identity: 
r(A + 1 + m)T(—m - A) 



7T 



("I) 



m+1 



7T 



sin(— 7r(A + m)) sin7rA 
Thus, all the statements of Lemma 6.2.2 are proved. 



□ 



The integrals in Lemma 6.2.2] do not depend on the choice of Li (i = 
1,2,3) as ordinary functions on t\ {0}. However, as we mentioned in 
Remark 16.1 .31 they depend on the choice of Li as distributions on M. because 
the poles of the distributions t± are located at A = —1, —2, —3, .... 
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To avoid this effect, we need to impose more constraints on the contours 
L{. Thus, let us assume further — 1 < s\ and — m — 1 < Sj (i = 2,3). 
Moreover, we assume — 1 < 7,- (J = 1, 2). That is, we shall assume from now 
that the integral paths Li (i = 1,2,3) are under the following constraints: 

- 1< 71, - 1< si < 0, (6.2.10) 

- 1 < 72, -m-l<s 2 <-m, (6.2.11) 

- m - 1 < s 3 < -m. (6.2.12) 



Then, the right-hand sides of (6.2.7)-(6.2.9) define distributions on R, which 
are independent of the choice of the integral paths (i = 1, 2, 3) subject to 
([6XTU1) -([6XT^. 

Proof of Theorem \6.2.1[ The first statement is a special case of Proposition 
[6X1(3). 

Let us show the second statement. The contour L used in (I6.2.2P and 
(|6.2.4p meets the constraints (6. 2 . lip and ()6.2.12p . and can be used as L2 
and L3. Likewise, the contour Lq used in (|6.2.ip and (|6.2.3h can be used 
as L\. Further, we shall assume that the contour Lq coincides with L when 
|ImA| is sufficiently large. 

The integrand of (|6.2.1|) has poles at A = — 1,-2, ... ,—m inside the 
closed contour Lq — L, and its residue is given by 

res r (~ A ) (2t) x - (~ 1 ) fc ~ 1 5<k-^(t) 

x=-kT(X + 1 + m) [ )+ ~ 2 k {m - k)\ [ ) 

for fe = 1,2,... ,m by (|7.1.ip . Therefore, by Cauchy's integral formula, we 
have 

1 ft A r ( -A) 



2irV^T VA JlJ r(A + 1 + m) 



III 



^2^(m-fe)! lJ 

as distributions. Hence, (6.2.5J is proved. 

Next, let us prove ([6.2.6). We recall from (|7.1.3p that the Laurent ex- 
pansion of the distribution t± at A = —k (k = 1, 2, . . . ) is given by 

*+- A + fc (fc-1)! 1 )+ + + ' 

(\ + k )(k-iy. [t) + t - + ■ 
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Combining with the Taylor expansions at A = —k (k = 1, 2, . . . , m): 

sinvrA = (-l) fc vr(A + k) H , 

tan7rA = 7r(A + k) + • • • , 
T(-A)2 A 



r(A + 1 + m) 



b + b 1 (X + k) + ■■■ , 



where 6q 



(fc-i)l 

2 fc (m-fc)! ' 



we have 



r(-A) 



(2$ 



T(A + l + m) \ tan(7rA) sin(7rAW 

6o((-l)* -1 tf (fc-1) (*) + (-l) fc <5 (fc_1) (0) 
7r(fc-l)! 



1 



(A + A;) 2 

6o(i; fc + (-l) fc C fe ) 6i((-l)^ 1 ^ fc - 1 )(t) + (-l) fc <5( fc " 1 )(i)) 



+ 



1 



7T 



7r(fc-l) 2 



A + fc 



+ 



(fc- l)\t~ k 1 
2 fc (m - /c)!vr A+T 



+ 0(1), 



as A tends to —A;. 

Therefore, by (|?TO]) and we have 



1 



27rV-l 



L 



r(-A) 



(2$ 



+ 



(2«)f 



T(A + 1 + m) \ tan(-7rA) sin(7rA) 



d\ 



E 



(fc-l)! 



' 2 k (m-k)\-K 



Hence, (6.2.6) is proved. Now, we have completed the proof of Theorem 

E2U □ 

Remark 6.2.3 (Bessel distributions). We shall use the symbols 

J m (2^2t^) = (2t) + 2 J m (2 v / 2l^), (6.2.13) 
K m (2y/2t^) = (2i) + 2 iC m (2 v / 2tT"), (6.2.14) 

m 

y m (272tT) = (2t)_ 2 y m (2 v / 2tl), (6.2.15) 

to denote the distributions defined by the right-hand sides of (|6.2.7[) — f)6.2.9p 
and by the contours Li (i = 1,2,3) satisfying (6.2.10)-(6.2.12), respectively. 
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It is noteworthy that J fn (2- v /2i + ) is a locally integrable function ont£M 
in view of the Taylor expansion (7.2.3), but K m (2^2t + ) andY m {2y/2~tZ) are 
not (see (|7.2.8|) and (7.2.9)/ Then, by the above proof of Theorem \ 6.2.1\ 
we have 



$+(t) = J m (2^) 

= (2t) + 2 J m (2y/2t^) 

*+(t) = J m (2^2T+ 



(6.2.16) 



m i i\fe-i 



(2t) + 2 J m (2 v / 2t7)-J] 



^ 2 fc (m - fe)! 

(_l)*-i 



J 2 fc (m - fe)! 



(6.2.17) 



~ 2(— l) m+1 ~ - 

*m(t) = y m (2 V / 277) + -i < K m (2 v / 2T) 



7T 



_m 2(— l) m+1 

(2t) + 2 y m (2 v / 2t7) + (2t)_ 2 K m (2^2t. 



7T 



(6.2.18) 



alternative way to define the distributions K m (2y/2t+) and Y m {2^/2~tZ) 
is to use the infinite sum expressions (|7.2.8|) and ([7.2.9ft . Let Q fc &e i/ie 
regularized distributions given by the second terms in (|7.1.3|) and (|7.1.4|) . In 
o/ ([7.2.8ft and (|7.2.9p . we define as distributions. 



Y m (2y/2t+) 



I m (k—lV 1 ~ 



7T " 2 fc (m - fe)! 



7T 



- - ( 2t +) Fj^TT)^ ' 

2(-l) m + 1 ~ , , -1 (_i)*(jfe_i)! i~ 

7T 7T 2 K {m—k)\ 7T 

--V(2t_)^ m + Z + 1) + V,(Z + 1) 



z=o 



Z!(m + Z)! 



// we set 

:=z ig ( _ 2t) ,«™+<+i) + w + i) 

7T ^- — ^ 



£=0 



l\(m + l)\ 



+ -(J m {2^j2t + ) log(2t+) + J ro (2 ^2t_) log(2t - 1)), (6.2.19) 

7T 
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then &m.(t) is a locally integrable function, and 

Y m {2^) + 2{ ~ 1)m+1 K m (2^2tl) = cj> m {t) - - V ( f t~ k . 

7T 7T L — ' 2 K (m — k)\ 

This gives another explanation of (|6.2.6p . In this book, we have adopted 
the integral expression (I6.2.3P for the definition of <& m (i) in place of the 
expansion (6.2.19). 

6.3 Differential equations for Bessel distributions 

The kernel K(x,x') of the unitary inversion operator ir(wo) is given by means 
of the Bessel distribution. In this section, we shall give a heuristic account 
on why the Bessel function arises in the kernel function. 

We begin with the observation that generic L + -orbits on C x C are 
of codimension one (see Lemma I6.3.2[) . This would force that the kernel 
function of any L + -intertwining operator (see ((2-5.6) for the definition of 
L + = M^^A) should be a function of one variable (x,x'). The second step is 
to make use of the differential equation arising from Ad(u>o)n max = n max (see 
(2.5.4)). Since n max acts on L 2 (C)°° as differential operators of second order, 
we get a differential equation of second order that the kernel distribution 
K(x,x') must satisfy (see Proposition I6.3.4"|) . The technical point here is 
that we have avoided using the Casimir operator of K because it acts on 
L 2 (C)°° as a differential operator of fourth order. 

The argument here was the clue for us to find a coarse form of the kernel 
K(x,x'). Though we did not use the results of this section for the actual 
proof of our main theorem, we think that the idea here is still helpful to 
get a coarse solution to a similar problem (see Problem 11.8.11 (2)) in other 
settings, namely, to find the integral kernel of the unitary inversion operator 
tt(wq) of the minimal representation of other groups. 

Let 6 : g i— > t g^ 1 be the Cartan involution of G. Since g £ G = 0(p,q) 

satisfies t gI Piq g = l PA where I PA = fa _| ^ J , we have t g~ 1 = I PA gI~^. 
Since wq = I p>q , we get 

9(g) =w gwQ 1 . (6.3.1) 
We let L + = M™ X A act on C by 

tJ-f f 

me ■ x = e mx 

for m £ M^ ax ~ 0(p - l,q — 1) and a = e tE £ A (see Section [231 for 
notation) . 
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Lemma 6.3.1. The kernel function K(x,x') of the unitary operator ir(wo) 
satisfies the following functional equation: 

K(x, x') = K(0{l)x, lx') for all I G L+. (6.3.2) 

Proof. Building on the unitary representation (ir, L 2 (C)), we define another 
unitary representation -k 9 on L 2 (C) by the following twist: 

n e (g) :=*(%)). 

Then, (|6.3.ip implies that n(wo) : L 2 (C) — ► L 2 (C) is an intertwining opera- 
tor from (tt,L 2 (C)) to (ir® , L 2 (C)) . In particular, we have, 

7T (/) o 7r(ioo) = tt(^o) ° tt(0 for any I G L + . (6.3.3) 

For I = m G M^ ax , we recall from (^XBl that 

(Tr(m)u)(x) = u^mx) for x G C. 

Hence, for any u G L 2 (C), we have 

/ K( t 6(m)x,a/)u(x')dn(x') = [ Kix^'^u^mx^dfxix"). 
Jc Jc 

Since d/i is M™ ax -invariant, the right-hand side is equal to 

K(x, t m~ 1 x')u(x')diJ,(x'). 

Since u is arbitrary, the kernel function must coincide: 

Jf(*0(?n)x,x') = K{x, t m~ 1 x l ). 

Replacing t 8(m)x with x, we have 

K(x,x') = if (mx,*m~V) for any m G M™ 8 *. 

Thus, (pOOl) holds for I G M™ ax . For I = a := e tH G A, we recall from 
(|2.3.12|) that 

(7r(o)u)(x) = e~ E± ^~ t u(e~ t x) (x G C). 
Since it (a) = 7r(a _1 ), the equation ()6.3.3p amounts to 

7r(u;o) = 7r(o) o 7t(tz;o) ° ir(a). 
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Hence, for any u G L 2 (C), we have 

f K(x,x')u(x')dfi(x') = e- ( P +q -^ t f K(e-*x,rE // )n(e-'rc")^(x // ). 
Jc Jc 

By the formula (2.2.3) of the measure dfi in the polar coordinate, we have 

dn(x") = e^-^d^x') for x' = e~V'. 

Thus, the right-hand side equals j c K(e~ t x,e t x')u(x')dfj 1 (x'). Hence, we 
have 

K(x, x') = K(e~ t x, eV) for any t G R 

and therefore 

K(x,x') = K(9(a)x,ax') for any a £ A 
Now, Lemma [6.3.11 is proved. □ 
Now let M™ ax act on the direct product manifold C x C by the formula: 

M max x C X C ^ C X C, (/, X, x') ^ (9{l)x, lx'). 

Furthermore, we define the level set of C x C by 

H t := {(x, x) G C x C : (x, x') = t} for t G R. 

Here, (•, •) is the standard positive definite inner product on W +q ~ 2 . Then 
we have: 

Lemma 6.3.2. 1) The level set Ht is stable under the M™ ax -action. 
2) Moreover Ht is a single M™ ax -orbit for any non-zero t. 

Proof. 1) For x, x' G R p+q ~ 2 (c W +q ) and I G M™ x ~ 0(p - 1, g - 1), we 
have 

(6(l)x,lx') = (wqIwq x,lx') 
= t x n lvj()lwQ 1 x 
= t x'woWQ 1 x 
= (x, x). 

Hence, ff t is M£ iax -stable. 

2) We replace (p — 1, q — 1) by (p, q), and consider the G-action on 

C := {(^O) ' ' ' j ^p+q-l) : x + " " " + x p-l ~ x p ~ ' ' ' ~ x p+q-l = 0} 
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in place of the M™ ax -action on C (this change allows us to use the notation 
iV max and iV max in Section [Oj) . Then, we recall from ([2X8]) that G acts 
transitively on C and the isotropy subgroup at eo + e p+q -\ = * (1, 0, • • • , 0, 1) 
is given by M™ ax iV max (see (121TH|) ). For t / 0, we take any (x,x') G H t . 
Then, we find gi £ G such that g\x' = *(1, 0, . . . , 0, 1). We write 9(gi)x = 
*(x , aVf-g-i) £le M p+9 " 2 © K. Then, 



t = (x, a;') = {9(gi)x, g x x') = x + x p+q . 



i- 



Let us consider the orbit of 6»(M™ ax iV max ) = M™ ax iV max on C. In view 
of (j2T3-6), we have 









nb j 


X 











f X0 + X ^- 1 f 26 b) V (6.3.4) 



.T 



xo — xwob 

X 

p+q -i + t xw^b) \ Q(b) J 



for 6 = • • • ,6 p+g _ 2 ) and x = t (x 1 ,--- ,x p+q - 2 ) G ^ p+q 2 ■ If x + 
x p+q _i / and t (x , • • • , x p+g _i) G C, we set 



-x 



xo + X p+q -i 
Since Xq — Xp +q _ 1 = —Q(x), we have 

b = so-sp+g-i g and Q(6) = x o - 

Then, we have 













raft j 


X 


\-\ 


0° 






\x P+q -i) 




\x p+q ~i) 






2(x + a; p +g-i) I _ 1 J 2 I 1 

Now, we set g := 6(nf,)gi. Then, 



(6(g)x,gx') = (n b 6(gi)x,0(n b )gix') = (- I 




Hence, the second statement is proved. □ 
Thus, we have the following proposition by Lemma 16.3.11 and Lemma 
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Proposition 6.3.3. K(x, ^OlcxCVffo ^ s °f the form 

K(x,x') = V({x,x')) (6.3.5) 
for some function ^f(t) defined on M. \ {0}. 

By lifting the inversion relation Ad(t/;o)n max = n max (see (|2.5.4p ) in the 
Lie algebra g to the actions on L 2 (C)k, we get the differential equation 
satisfied by ty. More precisely, 

Proposition 6.3.4. *&(t) satisfies the following ordinary differential equa- 
tion on R\ {0}: 

Proof. It follows from Ad(w )Nj = e^iVj (see (|2X3]» that 

7r(u;o) ° dn(Nj) = ejdir(Nj) o 7r(u>o). (6.3.7) 
We recall from Sections 12.31 and 12.4 that 

d7r(iVj) = 2v /Z T^ (see (|2.3.13|) ). 

*r(JVi) = ^^ e J P i ( see (12331!)), 

where -P, are the fundamental differential operators on the isotropic cone C 
such that 

d_ 

' dxj . 

if if; = ip\ c for a function defined in a neighborhood of C. Then, (6.3.7) 
leads us to the functional equation for any test function u(x'): 

*((x,a; / ))x / .u(x / )^(x / ) = / {Pj^({x,x')))u{x')dn{x'). (6.3.9) 
c Jc 

Now, in view of (|6.3.8p . 

P J -*«x,a/)) = e jXj Q(x')y"((x,x')) 

-4(2(x,x')^ // ((x,x')) + (p + g-4)^'((x,x / ))) 

= -^(2i>I/"(t) + (p + g-4)M/ / (t)) 

t=(x,x') 

Hence, the functional equation (|6.3.9p for any u G L 2 (C) implies that ^ 
satisfies the following differential equation: 

4* (*) = - (2m" (t) + (p + q- 4) V(i)) . 

Thus, Proposition 16.3.41 is proved. □ 



Pji>= (e jXj n-(2E+p + q-4)—^\ c (6.3.8 
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We write = t^. As 8 2 = t 2 ^ + t^, (^331) is equivalent to 

{8 2 + P + q 2 ~ 6 8 + 2t)V(t) = 0. 

Finally, let us see directly the differential equations that the Bessel distribu- 
tions $+(t), *+ (t), and $ m (t) (see (|6XT|) . (j6X2|) . and (^Ij) for definition) 
satisfy. It is easy to see that <&+(t), \P+(i), and x I' m (t) solve the following 
differential equation (in an ordinary sense) 

(9 2 + m8 + 2t)u = onl\{0}. (6.3.10) 

Of course, this fits well with what Proposition 16.3.41 asserts. On the other 
hand, the distribution 3> m (i) (see (|6.2.3p for definition) does not appear in 
the kernel function K(x,x'). We note that, as distributions, 

(o 2 + 2 m e + 2t)Y: 2 l { J_ k)l ^\t) = o, 

(6 2 + 2m6 + 2t) Y t~ k = - — - — — . 

v >^2 k {m-k)\ (m-1)! 

In particular, & m (t) (see ()6.2.3p ) does not solve (6.3.10), but solves the third 
order differential equation on K \ {0}: 

0{6 2 + 2m6 + 2t)$ m {t) = 0. (6.3.11) 
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Chapter 7 

Appendix: special functions 



We have seen that various special functions arise naturally in the analysis 
on the minimal representations. Some of their fundamental properties (e.g. 
integral formulas, differential equations, etc.) have been used in the proof of 
the unitary inversion formulas. Conversely, representation theoretic prop- 
erties are reflected as algebraic relations (e.g. functional equations) of such 
special functions. Further, different models of the same representation yield 
functional equations connecting special functions arising from each model. 

For the convenience of the reader, we collect the formulas and the prop- 
erties of special functions that were used in the previous chapters. 

7.1 Riesz distribution x\ 

A distribution fx on IR with parameter A £ C is said to be a distribution 
with meromorphic parameter A if the pairing 



is a meromorphic function of A for any test function ip £ Cq°(R). We say 
fx has a pole at A = Ao if (/a, f) has a pole at A = Ao for some cp. Then, 
taking a residue at A = Ao, we get a distribution: 



which we denote by res fx- 

A=A 

By Cauchy's integral formula, if C is a contour surrounding A = Ao, then 
we have 



C °°(R)^C, <p» TeB{f x ,<p) 
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and in turn we get an identity of distributions: 

res fx = \= f fx dX. 

A classic example of distributions with meromorphic parameter is the Riesz 
distribution x\ defined as a locally integrable function (and hence a distri- 
bution): 

x _ jx x (x > 0) 
lO (x < 0) 



.1, 



for A 6 C such that Re A > — 1. Then, x+ extends meromorphically to the 
entire complex plane, and all the poles are located at A = —1, —2, .... The 
residue is given by 



res x 



A 



•1) 



fe-1 



d^ix) (7.1.1) 



\=-k + (k-l)\ 

for k = 1,2,3, ... . Here, 5(x) is the Dirac delta function. To see this, we set 

T (0) 



f N (x) := <p(x) - 



< (k-l)\ 



x k ~\ 



Then, 



(4, y) = ^-1)] [ x x+k ^dx + £ x x m (x)dx 



/CO 
x <p(x)dx 



+ / x X (f(x)dx. 



The first two terms have a simple pole at A = — k with residue 



(fc-1)! \(*-l)! 

the second term is holomorphic if Re A > — N — 1 because ^pn{ x ) — 0(x N ), 
and the last term is an entire function of A because (£> is compactly supported. 
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Hence, (|7.1.ip is proved. Likewise, 

A j\x\ x (x < 0) 
10 (x > 0) 



extends a distribution with meromorphic parameter A and all the poles are 
located at A = 0, — 1, —2, .... They are simple poles with 

res X A = (7.1.2) 
x=-k (k-l)\ 

We write the Laurent expansions of x+ and x A at A = —k (k = 1, 2, 3, ... ) 
as follows: 



x x _ = —L-S^ 1 ) ( x ) + xZ k + (A + k)xZ k \ogx- + ■ ■ ■ . (7.1.4) 

Then, x+ k and xZ k are tempered distributions supported on the half lines 
x > and x < 0, respectively. We note that they are not homogeneous 
as distributions. Then, the sum x^_ + (— l) A x A becomes a distribution with 
holomorphic parameter A in the entire complex plane because 

res (xi + (-l) A x*) = 
x=-k 

for = 1,2,3,... . We now define a distribution 

*- fc :=(4 + (-l)V)| A= _ fc . ( 7 - L5 ) 

This distribution is homogeneous, and coincides with xZ. k + (— l) k x^_. 
For k = 1, x _1 is the distribution that gives Cauchy's principal value: 

(x- i ,cp)=nm( r + r) ^dx. 



e |0 \J _ OC) J e J X 

This formula is valid for any </? 6 Cb(K). Likewise, x _fc extends to a func- 
tional on the space Cq -1 (IR) of compactly supported functions on M with 



continuous derivatives up to k — 1. See the textbook [24] of Gelfand and 
Shilov for a nice introduction to these distributions. 
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7.2 Bessel functions J V ,I V1 K V ,Y V 

For Rez/ > 0, the series 

(-i)'(i) 2 ' 



=0 i!r(i + i/ + i) 

converges in the entire complex plane. Its sum J u {z) is called the Bessel 
function of the first kind of order v (see (T9j §3.54]). Jt,(z) extends mero- 
morphically to v G C by the Poisson integration formula: 



1 (z\ v 



and solves the Bessel differential equation: 

^ + ^ + (,W)). = 0. 
We set 

v ( s _ JjAz) COSVTT - J- U (z) 
SVO.VTT 

If v is an integer, say v = m, then this definition reads as 

J u {z)cos vk - J- U {z) 

Ym\Z) : — nm 

v—>m sill VTT 

Y u is known as the Bessel function of the second kind or Neumann's function. 
Further, we define two more functions by 

t/ — 1uTT -J — 17T 

I v (z) : = e 2 J v (e 2 z ) 



Both of them solve the following differential equation: 

.z 2 -^ + - (z 2 + z/ 2 )u = 0. 
dz z dz 

I v {z) is known as the modified Bessel function of the first kind, and is real 
when i/£l and z > 0. 
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K v {z) is known as the modified Bessel function of the third kind or Bas- 
set's function. Clearly we have 

K- V {z) =K v (z). 

We call J u , Y u , I u , and K v simply as J -Bessel, Y -Bessel, I -Bessel, and 
K -Bessel functions. 

The K-Bessel function satisfies the following formula (see |144 II, §7.11 
(22)]): 

(^) m (z- / K u (z)) = {-l) m z-»- m K u+m {z). 

This formula may be stated as 

(Id \ m ~ ~ 

in terms of the normalized K-Bessel function (|7.2.6p . By the change of 
variables z = 2e~ x , the m = 1 case amounts to: 

■^(e- ax K u (2e- x )) = - ae~ ax K y (2e~ x ) 

+ 2e- { - a+2)x K u+l {2e- x ). (7.2.1) 

The if-Bessel functions K u (z) reduce to combinations of elementary 
functions if v is half of an odd integer. For n € N we have 

\2z) 6 2L^j\(n-j)\(2z)3 
For instance, if n = 0, we have 

i 

vr \ 2 



The following renormalization is sometimes convenient: 

W : = (-r^) = X: j!r J^ + i) , (7.2.4) 

:= (7.2.5) 

:= (^)-^.(z). (7.2.6) 
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By the Taylor expansion as above, we see that both J u (z) and I v {z) are 
holomorphic function of z in the entire complex plane. 

J u (z) and Y u (z) are linearly independent of each other (whether v is an 
integer or not) and form a basis of the space of solutions to the following 
differential equation: 



d 2 u .„ ^.du 
z— + (2v + 1)— + zu = 0, 



dz 2 



dz 



(7.2.7) 



or equivalently, 



(9 2 + 2u9 + z 2 )u = 0, 
where 6 := z£. On the other hand, I v (z) and K v (z) solves 

(8 2 + 2u6 - z 2 )u = 0. 

In terms of Meijer's G-functions (Appendix I7.6|) or the Barnes hyperge- 
ometric function p F q (see (7.6.8)), we shall have the following expressions: 
(see (|7.6.12p - f|7.6.15l ): 





— ^02^ 


z 2 
4 




= <®( 


z 2 
4 


K u {z) 


" 2 02 





0,-1/ 



rVTi) oFl (^ 1; -T»' 



-Z/, 0, — 1/ 2 



0,-1/ . 



1 > 



The Mellin-Barnes type integral expression of J u (z), Y v (z), and K v (z) is 
also given in Lemma 6.2.2, 

For m = 1, 2, 3, ... , the infinite sum expressions of Y m {z) and K m (z) (or 
Y m (z) and K m (z)) at z = are given in [TJl II, §7.2, (31) and (37)], which 
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may be stated as follows: 



1 m 

^)=-~£(f) 



-2k (fc-l)l 

tt^ v 2' (m-A;)! 

2 ~ z 
+ -J m (z)log(-) 

7T Z 



7T ' 2 t! m + / ! 



^ m 



*\_ 2 * (fc-l)! 



2^ v 7 y 2' (m-k)\ 
k=i y ' 

+ (-ir +i 7 m ,(z)iog(|) 

+ 2 (_1) A. ( 2 } ^TTT)! • (7 - 2 - 9) 

Here, the function ip(z) is the logarithmic derivative of the gamma function: 

dlo g r(z) T'(z) 



1>(z) :-- 



dz T(z) 



The i/j function is meromorphic with simple poles at z = 0, — 1, —2, .... 
Next, we summarize the asymptotic behaviors of the Bessel functions: 

Fact 7.2.1 (see [U Chapter 4], [79j Chapter VII]). The asymptotic behaviors 
of the Bessel functions at z = 0, oo are given by 

1) As z tends to 0, J v {z),I v {z) = 0{z u ). 

For v > 0, 

Kv{z) = ^( Z -)~ 2V + o{z- 2 ») asz^O. (7.2.10) 



2 V2 
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2) As z tends to infinity 



Mz) ~ V ^l C0 < 2 " T " i) g — (2ipr- 

L) J >,2j- 
(2z)2j'+ 1 



00 



z ~ T ~ l) S ( ^ ) (Iarg.|<vr), 

3=0 

j=0 



W v 7 ^^ 2z J ^ 2z J V 2 6 2 J 

3=0 j=o 

j=l 

In particular, we have 

K u (2z) = ^e~ 2z z~ v -^ (1 + 0(1)) as z^ 00. 
Here, we have used Hankel's notation: 



(«,J) := ("IF- 



(4a 2 - l 2 )(4a 2 - 3 2 ) • • • (4a 2 - (2j - l) 2 ) 
2 2 ij! 

Finally, we list some integral formulas for the Bessel functions: 

Bl (the Mellin transform of i^-Bessel functions, see [26|, p. 684]). For Re(/x+ 
1 ± v) > and Re a > 0, 

Jo 11 
Equivalently, we have 

t s K v {at)dt = 2 s - 1 a- s - 1 r(^-)r(^- - !/). (7.2.11) 
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B2 Formula of the Hankel transform due to W. Bailey [3] (see also \15\ 
§ 19-6 (8)]). 

/"OO 

/ t x ~ l J^{at)J u {bt)K p {ct)dt 
Jo 

_ 2 x - 2 aWT{\{\ + /i + v- p))r(i(A + fi + v + p)) 



c x +v+»r(n + i)r(i/ + 1) 



11 a 2 b 2 

x F 4 (-(\ + p + v - p), -(\ + p + v + p); p + l,v + 1; - — , - — ). 

(7.2.12) 

Here, -F4 is AppelFs hypergeometric function of two variables (see (17.7.40 ). 
B3 (see [13 §7, 14.2 (36)]) For Re(a + 0) > and Re(p ± /* ± 1/ + 1) > 0, 

/•oo 

2 P+ 2 r(l-p) / K^(at)K u ((3t)r p dt 
Jo 



.1 + 1/ + // — p 1+u—p—p (3 



a" 

l+v+p- p l + i/ - p, - p + - p p. - p 

{ 2 ) \ 2 ) { 2 ' ^ 2 

In particular, we have 

"K fW ^= ^'^'-"\ (7.2,3) 



7.3 Associated Legendre functions 

The associated Legendre functions on the interval (—1,1) is defined as the 
special value of the hypergeometric function: 

The associated Legendre functions satisfy the following functional relation: 

jL((l-a?)-$PM-xj) =(l- x 2 y^P£+\- x ), (7.3.2) 

which is derived from the following recurrence relation (see [26} §8.733 (1)]): 

(1 - x 2 )^-P?{x) = -Vl-x 2 P<? +1 (x) - pxP^(x). 
ax 

Integral formulas for the associated Legendre functions: 
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LI (see [26j p. 803]) Formula of the Riemann-Liouville integral: Re A < 1, 
Re// > 0, < y < 1, 



i r y 



f V {y-xY~ l {x{l-x)) *P*{l-2x)dx=ty{l-y))> h^(l-2y). 
Jo 

(7.3.3) 



L2 (see [261 P- 798]) For 2 Re A > | Re// 



7r2T(A + f)r(A- f 

r(A + 1 + i)r(A-f)r(^±^)r(- 



(l-arr~ L Pjf(x)dx - 

(7.3.4) 



7.4 Gegenbauer polynomials 

Definition of the Gegenbauer polynomials: For I 6 N, we define 
C*( X ) - ^ r(2/i + tllilz " 2 ) l ~" fq (741) 

Mx) - 2' r(2/x)r(/, + z + i) /! x) J " (7A1) 

Slightly different from the usual notation in the literature, we adopt the 
following normalization of the Gegenbauer polynomial: 

C?(x):=rtoC?(x). (7.4.2) 
By using Gauss's duplication formula 

r(2/,) = 2 2 ^V"ir(//)r(/, + I), (7.4.3) 

the definition (I7.4.ip may be stated as 

C?(x) = (-1)^ + /)^ _ , 2) -, + |^ ((1 _ JyH-ky (7.4.4) 



The special value at /x = is given by the limit formula (see [141 §3.15.1 
(14)]): 

Q°(cos 9) = lim r(/j)Cf (cos 6) = 2cos ( W \ (7.4.5) 
On the other hand, the special value at I = is given by 



c£(x) = r( M ). 
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Connection with Gauss's hyper geometric function (see |14^ §3.15 (3)]): 

lK ' r(/ + i)r(2/i) v p p 2' 2 ; 
rq + 2/x)r(/x) F , z + 2/* / i 2 
= r(f + i)r(2^) 2Fl( -2-' " 2' " + 2 ' 1 " x Y (7A6) 

Here, the second equation is derived from the formula of quadratic transfor- 
mation for hypergeometric function (see |14^ §2.11 (2)]): 

1 1 

2 F 1 (a, b;a + b+ -; 4x(l - a?)) = 2-Fi(2a, 26; a + 6 + -; x). 

By using Rummer's transformation formula for the hypergeometric func- 
tions: 

2 F 1 (a,p;r,z) = (l-zy- a ~' 3 2 F l (7 -a, 7-/?; 7; *), 

one can obtain the following relationship between the Gegenbauer polyno- 
mials and the associated Legendre functions. 

Ct{x) = f ^ + 1 \ (1 - s»)M , (x), -Kx<l. (7.4.7) 

2^~2r(/ + i) 

Integral formulas for the Gegenbauer polynomials: 
Gel (Orthogonality relations; see [TH §3.15.1 (17)]) For Re fi > — |, 

1 ~, 1 fO tfl^m, 

Ct(x)C^(x)(l - x 2 r~-2dx = { 2 i- 2 ^ r(t+2M) _ (7.4.8) 



Ge2 (see [13 §7.321]) For R e/ u > -\ 



L 



2- 

\i - xr-h^c?(x)dx = n2 ^% +l) ^JM-y 

(7.4.9) 



Ge3 (see 03 Lemma 8.5.2]) For a G C, Rei/ > -1, and / G N, 



J v (ay/x + T)C l 2 {x){l + a?)*(l - x)"dx = 

2l(-i)-^r(2, + i + 1 ) Wi(V , a) (7410) 
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Ge4 For Rev > —\ and Re A > -1, 



x + y) x Cl(y)(l-y 
^T{2u + fc)r(A + 1) 



2\v- 



<dy 
x 2 ) 



"2 + 4 P 



u+k- 



(-x). (7.4.11) 



This formula (|7.4.11j) is essentially the integration formula (|7.3.3p for the 
associated Legendre functions. For the sake of completeness, we give a 
proof: 

The left-hand side of (I7.4.11|) 

^ r(2iy + fc) f \i-y 2 )^Hx + y) x pt: i(v)dy by CSZD 



r~lk\ J-x v+k ~ 



>A+1 



2^(2, + *)r(A + l),l-x^l + M p - t7 l ( _ i) by 



fc! V 4 / 

= the right-hand side of (I7.4.1ip . 

7.5 Spherical harmonics J-C 3 (M. m ) and branching laws 

A spherical harmonics f of degree j = 0,1,2,... is the restriction to the unit 
sphere S m ~ l C W 71 of a homogeneous harmonic polynomials of degree j in 
M m . Equivalently, / is a smooth function satisfying the differential equation: 

A sm -if = -j(j + m-2)f. 

The space of spherical harmonics of degree j is denoted by 

W(R m ) := {/ G : Agm-if = + m - 2)f}. 

When m = 1, it is convenient to set: 

^(M 1 ) : = CI, IK 1 (M 1 ) := Csgn, W (M 1 ) := (j > 2). 

The following facts are well-known (see [311 Introduction], [69]): 

HI For / G ^(M m ), /(-x) = (-l)V(x). 
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H2 0(m) acts irreducibly on W(R m ). 

H3 IK^M™) is still irreducible as an S'0(m)-module if m > 3. 

H4 W(R 2 ) = Ce^ e Ce-^ 9 , j > 1 as SO(2)-modules, where = 
tan" 1 !, G R 2 . 

H5 ^'(K m )| } ~ 0| =o ^{'(l" 1 " 1 ) as 0(m - l)-modules. 

H6 The Hilbert space L 2 {S m ~ 1 ) decomposes into a direct sum of the space 
of spherical harmonics: 

oo 

L 2 (S' m ~ l ) ~ J]®^(M m ). 
j=o 

Here, > stands for the Hilbert completion of the algebraic direct 
sum0°l o ^'(R OT ). 

Let (xo, G M m , x G M m_1 be a coordinate of M m . Then, the branching 
law H5 is explicitly constructed by the 0(m — l)-intertwining operator 

as follows (see [69j Chapter III]): 

Fact 7.5.1. For < i < j and <f> G 3<*(IR m_1 ), we define a function I^j<i> 
on S m ~ l by 

oJCj'-i (^o)- (7-5.1) 

Here, C"{z) is the normalized G eg enbauer polynomial (see (|7.4.2ft ). Then, 

1) /™j(0) G ^'(IR m ). 

2) I£L>j gives an injective 0(m — l)-homomorphism from IK*(M m_1 ) to 
W(R m ). 

3) (L 2 -norm) 

II/™ (6)\\ 2 23 " m " 2i ^(m-2 + i + j) 2 
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Proof. We use the following coordinate: 



[-l,l]x S m ~ 2 ^ S m ~\ (r,ri)^u> = (r,Vl-r 2 ri). (7.5.3) 
Then, the standard volume form du> on the unit sphere S m ~ l is given by 

q m — 3 

(1 — r ) 2 drjdr. Therefore, 



11/^.(0)11!^-!) = J^J sm Jl- r y\<f>( V )f\C^ + \r)\\l-r 2 )^d V dr. 

Now, apply (T7A8]) . □ 

We illustrate the intertwining operator by the two extremal cases, 
i = and i = j: 

Example 7.5.2. 1) The case i = 0. Then, 

(/ %l)(x ,x) = C^- 2 (x ) (7.5.4) 

is £/ie generator of 0(m — l)-invariant vectors in 5f J (M m ), where 1 is £/ie 
constant function on S m . 

2) TTie case i = j. Then, we have simply 

I^M)(x ,x) = T(m)\x\U(^j). (7.5.5) 



J— * J ' ' \x\ 



7.6 Meijer's G-functions G™jp(x 



Let m, n,p and q be integers with < m < q, < n < p and 

c := m + n > U. 

2 ~ 

Suppose further that the complex numbers a±, . . . ,a p and b\, . . . , b q fulfill 
the condition: 

a, -b k 7^ 1,2,3,... (j = 1, . . . n; fc = 1, . . . ,m). (7.6.1) 
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Then, Meijer's G-function of order (m,n,p,q) is denned by the Mellin- 
Barnes type integral (see O §1.19, §5.3], [22 I, §1], [601 §8.2]): for x > 0, 



/~im,n 
p,q 



ai, 



2tta 



nr(6 J -A)nr(i-a J + A) 

3=1 3=1 



ft r(i-6 i + A) f[ r( 0i 

j=m+\ j=n+l 



-x x d\, (7.6.2) 



A) 



where an empty product is interpreted as 1. 

The contour L starts at the point 7— \J — loo (7 is a real number satisfying 
(|7.6.4p below if c* = 0), leaving all the poles of the integrand of the forms 

A = bj,bj + 1, 6j + 2, . . . (l<j<m) (7.6.3) 

to the right, and all the poles of the forms 

A = Oj — 1, aj — 2, ay — 3, . . . (1 < j < n) 

to the left of the contour and finishing at the point 7 + \J — loo. 
Here, the condition on the real number 7 is given by 

(q-p)"f > Ren, (7.6.4) 

where we set 

q p 

P- 



3=1 



v 

3=1 



+ 1. 



It follows from the asymptotic behavior of the gamma factors (see Lemma 
6.1.4) that the integral (7.6.2) converges and is independent of 7 if one of 
the following conditions holds: 

1) c* > 0, I argx| < c*7r; 

2) c* > 0, I argx| = c*7r, (q — p)*y > Re fi. 

In particular, the G-function extends holomorphically to the complex 
domain | argx| < c*tt if c* > 0. 

The G-function is symmetric in the parameters ax, 
a n+ i, ...,a p , in 61, ... , b m , and in b m+1 , . . . ,b q . 

Obvious changes of variables in the integral give 



, o n , likewise in 



S(-im,n 



f-iin,n 



X 



h, 
ax, 
bx, 



f~im,n 
PA 



f-m,rn 
q,p 



ax + S, 

bx + s, 

1-bx, 
I - ax, 



bq + S 
l-b q 

1 — a T 
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The G-function G™^ \ x 



(see mi §5.4 (1)]): 



&ij • • • j p \ sa tf s ff es ^he differential equation 
b 1 ,...,bg l 



v , i 



^_ l)P - m -n x ^ {x _ _ a . + Y[( x — - hl ))u = 0. (7.6.5) 
j=i 3=1 
If p < g, the only singularities of (|7.6.5p are x = 0, oo; x = is a regular 
singularity, x = oo an irregular one. For example, GqI(x \ bi,b2,bs,b^) 
satisfies the fourth order differential equation: 

j=i 

The condition (7.6.1) implies that none of the poles of T{bj — A) (j = 
1,2, ... , m) coincides with any of the poles of T(l — a k + A) (k = 1, . . . , n). 
Suppose further that 

b j -b k ^0,±l,±2,... (l<j<k<m). 

Then the integrand (as an ordinary function for x > 0) has simple poles 
at the points (|7.6.3p . (We note that as a distribution of x, x x has simple 
poles at A = —1, —2, —3, . . . , and the analysis involved is more delicate; see 
Sections 16.11 and l6.2i ) For p < q, by the residue calculus, we obtain (see [551 
I, (7)]): 



affix 



oi, • • • , a ? 
h, • • • , b Qj 

m n 

nr(6 J --6 fc )nr(i + 6fc-o i ) 

m J=l 3=1 

= ^ ^ - x bk 

k=i n r(i + b k -bj) n r(a, -6 fe ) 

j=m+l j=n+l 

x p F,_i(l + 6 fc - ai, . . . , 1 + b k - a p ; 1 + b k - h, ■ ■ ■ , 1 + b k - b q ; (-l) p - m " n x). 

(7.6.7) 

Here, p F q denotes the Barnes generalized hypergeometric function: 

v 

^ x k n + 

3=1 



p F q (ai, ■ ■ ■ ,a p ;Pi,. . . ,P q ',x) = ^ 



k=o k\ f[ /3 j (^ + l)---(/3 j +^-l) 

(7.6.8) 
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For example, 2F1 («i , «2 ; Pi ] x) is the Gauss hypergeometric function, and 

00 u 
oFi(0; x) = g fc! ^ + 1) ... 03 + A ._ 1) - 

Similarly, for q < p, if a,- — a/% 7^ 0, ±1, ±2, ... (1 < j < k < n), we have 



pira.n I 



a%, ■ ■ ■ , a p 

£>1, • • • , 



I] r(o fc - ty) II r(6j - a k + 1) 
i=i j=i 

E 



fc=i f[ r( ai -a fc + i) f[ r(a fe -6 i 

j=n+l j=m+l 



k 

x gFp.^l + 61 - a k , . . . , 1 + b q - a k ; 1 + a x - a k , - : - , 1 + a p - a k ; (-l)^ m ~ n 

(7.6.9) 

For p < q, it follows from (|7.6.7p that 

GTa n ( x '"' M = 0(|x| min ( Rcbl '-' Rcbm) ) (7.6.10) 

as x -> (see also [HI I, §5.4.1 (8)], but there is a typographical error: 
max Re 6^ loc. cit. should be minRefe/j). On the other hand, the asymptotic 
expansion of Gp] q n {x) (p < q) for large x > that we need in this book is 
the following case: 

Fact 7.6.1 ([551 VII, Theorem 17]). Let m,p and q be integers satisfying 

< p < q — 2 and p+l<m<q — 1. 

Then the G -function G™g°(x) possesses the following asymptotic expan- 
sion for large x > : 



G™<°(x) ~ ^ m 'X 9 (xe^- m )^) + A m >° q H p , q (xe 



Here, H Ptq (z) is a function that possesses the following expansion (see 
SSI I, (25)]): 

f, a / (2vr) 2 ^ i Mi M 2 \ 
flp,,(z) = exp Up-qjzi-Az 6 y ' +^ + ^ + --- , 



Vq-p 



Zl-P Z1~P 



148 



where Mi, M 2 , ■ ■ ■ are constants, and 9 is given by 

q p 



1 (p — q+l 



q-p 



3=1 3=1 



3 ' 



I, (23)]. 



The coefficients A m 'q and A m 'q are given by 



A m,u ._ (_ 2vr ^rT)™-9 e -(Wi+-+MTV=i ) [551 II, (45)], 
A m '° q := (2vr^/^T) m ^« e (Wi+-+»«)^ ) [55l II, (46)]. 

Example 7.6.2. For (m,n,p,q) = (2,0, 1,3), c* = 0. We take 7 such that 

7 > - Re(6i + 6 2 + 6 3 - ai). 



Then, we have an integral expression: 



r 13 



a-i 



1 



r (fel -A)r (62 -A) xA ^ 



bi,b 2 ,b 3 J 2ir^l J l r(l - 63 + A)r(ai - A) 



where the integral path L starts from 7 — v^Too, leaves 61,62 to toe ng/i£ 
and ends at 7 + V — loo (see Figure [7. 6. i| ). 
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Example 7.6.3. If p = 0, the G-function is denoted by G^' {x \ b\, . . . , b q ). 
The G-function that we use most frequently in this book is of type Gq®. 
Again, we have c* = 0. Then, we have an integral expression: 

,20^ ,^ _ i f r(&i — A)T(6 2 — A) , 



G&(x | h, 62,63,64) = — = / rn V-u^n h M\\ x dX ' 

27rv-i Jl r(i - 63 + A)r(i - o 4 + A) 

where L starts from 7 — y 7 — loo, leaves b\, 62 ^° ri^/ii, and ends at 
7 + V - loo ( see Figure [7. 6. 1\ ) for 7 G K such that 

7 > -(Re(6i + 6 2 + 63 + 64) - 1). 

In Section \A.5\ we need the following lemma on the asymptotic behavior: 

Lemma 7.6.4. The asymptotic behavior of the G-functions Gq®(x \ 61, 62, 1— 
7 — 61, 1 — 7 — 62) o,re given as follows: 

1) As x tends to 0, G 2 ° A (x \ b u b 2 , 1 - 7 - h, 1 - 7 - 6 2 ) = 0(x min ( bl ' fc2 )). 

2) As x iends to 00, 



Ox I 61,62,1-7-61,1-7-62) 

i ir i cos(4x2 - (7 + 61 + 6 2 + 7 )7r)(l + Pix"^ + P 2 x _1 + 



1-47 

2; 



'2tt 

1 — 4"C 1 13 

+ x~s~ sin(4x4 - (7 + 61 + 6 2 + -)7r)(Qix"4 + Q 2 x~i H ). 

(7.6.11) 

.Here, Pi, • • • , Qi, • • • are the constants independent of x. 

Proof. 1) This estimate is a special case of (|7.6.10p . 
2) We apply Fact 17.6.11 to the case 

(m,p,q) = (2,0,4), (61,62,63,64) = (61,62, 1-7-61,1-7 - 6 2 ). 
Then, the coefficients A 2 ®, A 2 ® and the constant 6 amount to 

420 \_ (2 T -2+&i+6 2 )7iV=T 120 \_ p -(2 7 -2+fei+b2)7r v / ^T 

A 4 ~ 4^2 e ' 4 ~ 4vr 2 

The expansion of -Ho, 4 (xe =t27rv/ ~ 1 ~) is given by 

H 0A (xe± 2 ^) = e ^ Li f^^ f Ml ± -J*_ + ...). 

V 2 \/— lx 4 ) 
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Hence, Gq®(x) has the following asymptotic expansion: 
_ _L e -(4^- 7 r( 7 +6 1 +6 2 -|))V=T x i^ / (27T)I Mi 

_ J_ e (4 :K 4- 7 r( 7 + 6l+62 -|))^ x ^ ( (27[)l _ Mi \ 

which is expressed as the right-hand side of (I7.6.1ip by virtue of the formulas 

gCTTv^T + g-CTTv^T = 2 cog ( c7r ) and gCTTv^ _ g-CTT^ = S hl(c7r) . □ 

Finally, we list the reduction formulas of G-functions that are used in 
this book: 

G$j(x | a,b) = x^ a+b) J a ^ b (2x^) pU §5.6 (3)], (7.6.12) 
G$(x | a, 6) = 2x^ a+b ^K a _ b (2x 1 2) [H §5.6 (4)], (7.6.13) 

Gl 4 (x\a,a+^b,b + ^) = x^ a+b U 2{a ^ b) (Axi) [HI §5.6 (11)], (7.6.14) 
a ~ 1 l) = ^ (a+6) n-a(2x5) mi §5.6 (23)]. (7.6.15) 



7.7 Appell's hyper geometric functions Fi, F2, F3, F4 

Appell's hypergeometric functions (in two variables) F±, F2, F3, F4 are de- 
fined by the following double power series: 

F 1 (a,P,P', T ,x,y):= £ ^]^ )m ^ )n x m y n , (7.7.1) 

m,n=0 

F M ,^>;*,V):= £ ^^ T, (".2) 

m,n=0 

F 3 (a,ct,l3,l!f, r ,z,v):= £ W^m^V ^ (7 . 7 . 3) 

m ^l (7) m+ nm!n! 

F 4 (a,^,7,7 ;x,y := > —a; y. 7.7.4 

m,n=0 

Apl Reduction from F 3 to Fi ([H §5.11, (11)]): 

F 3 (a, a + c/; X, y) = (1 - y)^ 13 ' Fi(a, /5, /?', a + a'; x, 



(7.7.5) 
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Ap2 Reduction from F 3 to 2 -Fi (d §5.10, (4)]): 

F 3 (a, 7 - a, (3, 7 - /3; 7; x, y) = (1 - 2/) Q+/3 ~ 7 2 i ? i(a, 0,y,x + y - xy). 

(7.7.6) 

Ap3 Reduction formula of F 4 (pU §5.10, (8)]): 
F 4 (a,/3;l + a-/3,/3; 



(l-x)(l-y)' (l-x)(l- 2 /) / 

= (1 - iOVifo /3;l + a-/3; "f 1 "^ ). (7.7.7) 

1 — x 

Ap4 Single integral of Euler's type for Fi ([3 §5.8.2, (5)]): For Re a > 
and Re(7 — a) > 0, 



Fi(a, f3,f3' , 7; x,y) = 



r(a)r(7 - a) J 



u a_1 (l - u) 7-0 - 1 ^ - nx)- /3 (l - uy)-P'du. 

(7.7.8) 



Ap5 Double integral of Euler's type for F 3 (flU §5.8.1, (3)]): 

r(7) 

F 3 {a,ct,fl t p,r,x,v) = mm)T{l 

x // u/3 ~ lv/3 '^ 1 ( 1 - u - u ) 7 ~ /3 ~ /3 '" 1 ( 1 - n2; )^( 1 -' t; y)^ Q ' dnd ' t; 

Re /? > 0, Re /?' > 0, Re( 7 - (3 - f3') > 0, (7.7.9) 
where D := {(u, v) e M 2 : u > 0, u > 0, u + v <1}. 

7.8 Hankel transform with trigonometric parame- 
ters 

This section presents an integral formula (|7.8.ip on the Hankel transform 
with two trigonometric parameters. In the conformal model [U] (i.e. the 
solution space to the Yamabe equation) of the minimal representation, in- 
finite vectors can be explicitly expressed in terms of spherical harmonics 
(e.g. Gegenbauer's polynomials). On the other hand, in the L 2 -model (the 
Schrodinger model) which is obtained by the Euclidean Fourier transform 



152 



of the conformal model (or the iV-picture in a terminology of representation 
theory), it is not easy to find explicit K-Rmte vectors. The formula (7.8.1) 
bridges these two models and gives an explicit formula of if-finite vectors 
in the Schrodinger model (see the proof of Lemma I3.4.4|) . 

Since we have not found this formula in the literature, we give a proof 
here for the sake of completeness. The method here is a generalization of 
the argument in [H2 §5.6, 5.7]. 

Lemma 7.8.1. The following integral formula on the Hankel transform 
holds: 

" r» Jf f-^f-j) j, km* 

o \ cos 9 + cos <p J \ cos 6 + cos <p ) 

r(/x- v + 1) (cos 9 + cos 0) sin" flsin^ C^tl (cos <f>). (7.8.1) 



7T 



Proof. By Baily's formula ()7.2.12p of the Hankel transform, the left-hand 
side of (I7.8.ip equals 

T(n + v + l) 2^ sin^ 9 sin u <j> 



T(v + l) (cos 9 + cos 4>Y l+L/ 

„ , , sin 9 x9 . sin 
x F 4 (^ + l,fi + u + l-n + l,u + l;-( — -) 2 ,-( 



cos 9 + cos q> cos 9 + cos <p 

(7.8.2) 

Here F4 denotes Appell's hypergeometric function (see (|7.7.4p ). Thus, the 
proof of Lemma 17.8. II will be completed if we show the following: 

Claim 7.8.2. We have 

„ , , sin (9 n9 , sin 6 
F i (n + l,n + u + l; f i + l,i/ + 1;- ^- 2 ,- V 2 

COS P + COS cos y + COS 

_ (cos a + cos r&.-, + i)r(„ + Dyj (co .^ (7 . 8 . 3) 
2^-^+i^/i r(/x + i/ + i) ^~ uK V} y ' 

□ 

Claim 17.8.21 is essentially a restatement of |49l Lemma 5.7]. For the 
convenience of the reader, we include its proof here. 

Proof of Claim {T.8.2 . We recall a quadratic transformation for hypergeo- 
metric functions (see p~H § 2.11 (32)]): 

fy fy _|_ 1 _ Oft —A-7 

2 F 1 (a, 0; 1 + a - (3; z) = (1 - z)- a 2 F l {-, —— ^; 1 + a - ft 



2' 2 ' ' " (1-z) 2 '' 

(7.8.4) 
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Combining the reduction formula (|7.7.7p with ([7.8. 4), and using the sym- 
metry of a and b; (c, x) and (d, y) in F^a, b; c, d; x, y), we have 

— x — y 

F 4 (a, /3; a, 1 - a + /?; — -, — r) 

(l-a?)(l-y) (l-x)(l-y) 

/ (i- g )(i-y) ^ ft l-2a + /3 4 y(l-x)(l-y) ^ 

= 1^ l-*y ) 2Fl( 2' 2 + (1-^ } - 

(7.8.5) 

Consider the change of variables from (x, y) to (0, 0) by the following iden- 
tities: 

sin 9 \ " y / sin 



(1 — x)(l — y) \cos 9 + cos <^> / ' (1 — — y) \ cos 9 + cos 

such that (x,y) = (0,0) corresponds to (9, (f>) = (0,0). Then, a simple 
computation shows 

1-xy 2 4y(l-x)(l-y) . 2 

= sin m. 



(l — x)(l — y) cos# + cosc/>' (1 — xy) 2 

Now, we set 

a = fj, + 1, /3 = n + u + l, 
in (|7.8.5ll) . Then, the left-hand side of (|7.8.31) amounts to 

2 FW , — ;z/ + l;sm 0). (7.8.6) 



s cos 9 + cos <fi ) 2 
By using (j7.4.6[) . (]7.8.6p is expressed as 

cos 9 + cos <j) \ ^ +u+1 T(p-u + l)T(2v + l) 



\ ^ r(^-, + i)r(2, + i) ~, + i 
J r(/^ + 1/ + i)r(r/ + i) ^ ( 



By using Gauss's duplication formula (7.4.3), we get Claim. □ 



7.9 Fractional integral of two variables 

In Section 15.31 we find explicit eigenvalues of intertwining operators on 
L 2 (S P ~ 2 x S q ~ 2 ) based on the Funk-Hecke formula (see Example I5.3.2p . 
The following lemma is the substantial part of computation there. 
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Lemma 7.9.1. For Re fi, Re v > — \, Re A > — 1 and l,k GN, we Ziaue i/ie 
following formula for the fractional integral: 



(7.9.1) 



62 1 ~ A r(A + ji + v + 1) 

~ p^ A+2^+2i/+;+fc+2 ^ A+2/j+Z-fc+2 ^ A+2;y-Z+fc+2 jp^ \-l-k+2 ^ ' 

where 

(±i) i+fc 7r 2 r(2/i + z)r(2i/ + fc) 

is a constant independent of X. 

Proof. The left-hand side of (|7.9.ip amounts to 

'i ( /I rilTiy^ (y)(1 " y 2 r ^) Wt 1 - * 2 ) M ~^ 

= k) f (a - x 2 )f + f + ip; + A ;;^(-x))cf(x)(i - X ^-Ux 

2 -"-2M-i+i T r(2i/ + fc)r(2M + z) 2 ,a +m+ ^_i_i -A-v+f-i, 

= — t / (l-x z ) 2 iP 2 (-x)dx 

_ 2 1 - A - 2 ^ 2 ^r(2 / u + l)T(2v + fc)r(A + /j + v + 1) 

~~ fcl p ^+2M+2i/+f+fc+2 ^p^ A+2/j+;-fc+2 ^p^ A+2v-Z+fc+2 ^ A-Z-fc+2 j' 

Hence, the right-hand side of (j7.9.ip follows. Some remarks on each equality 
are given in turn: 

First equality follows from Gc 1 in Appendix I7.4I 

Second equality. First, we made use of the integral by parts because 
we have (see ()7.4.ip ) 

cf (sxi - x^y-i = <L^, r ^ +0 ^ ((i - * 2 r +l ^) ■ 

iVA ; 2 2 »+ l ~ l l\ T(p + I + \) dx l V 1 ) 

Then, we applied the functional relation 

(ft ( i - 9\ ^j-"^ 1 x-. - A— V— h , s\ /„ Oi 1 4- 1 t-. — A — v-\-l — h / \ 

s ((i-^ ,+i+4 V}H = (1 ^ )j,1< Vj 

which is obtained by iterating (7.3.2). 

Third equality. We applied the integral formula (|7.3.4h after changing 
the variable x t— > —x. 

Therefore, the proof of Lemma l7.9.1l is completed. □ 
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